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ABSTRACT 


Recent  studies  and  surveys  have  concluded  that,  in  general,  the  docu- 
ments produced  to  support  the  understanding  and  use  of  computer  models 
are  inadequate.    This  paper  describes  the  issues  and  concerns  of  compu- 
ter model  documentation  and  proposes  an  approach  for  the  development 
of  adequate  documentation.    First,  a  number  of  documentation  studies 
and  reports  are  reviewed,  including  software  documentation  guidelines 
and  model  documentation  procedures.    Then,  based  on  the  relationship 
between  the  phases  of  the  model  life  cycle  and  documentation  information 
needs,  a  set  of  documents  is  proposed  and  described. 

The  author  takes  a  highly  critical  view  of  the  past  and  present  inade- 
quate state  of  documentation  procedures  for  computer  models.  The 
attention  of  computer  model  sponsors  and  developers  must  be  directed 
to  this  area.    Otherwise,  the  author  feels,  there  will  be  an  unfortunate 
decline  in  the  use  of  decision  models  as  aids  in  the  analysis  of 
important  policy  issues.    The  course  of  action  recommended  in  this  report 
is  an  extreme  position  as  to  the  total  information  and  number  of  docu- 
ments required  to  produce  adequate  documentation.    The  author  calls  for 
the  capturing  of  all  information  generated  during  a  model's  life  cycle. 
Further  research  is  needed  to  adapt  this  extreme  position  to  the  reali- 
ties of  cost,  resources,  model  complexity,  and  model  use. 


Key  words:  Computer;  computer  model;  documentation;  documentation 
procedures;    model;    model  doc\imentation. 


NOTE: 


This  document  is  the  final  report  of  a  study  conducted  to  provide 
background  and  planning  information  for  a  proposed  program  to  develop 
standards  for  the  management  of  government  modeling. 

As  a  research  result,  it  is  published  in  its  entirety.  NBS  neither 
accepts  nor  rejects  the  recommended  approach  to  model  documentation 
contained  herein. 

Paul  F.  Roth, 

NBS  Institute  for  Computer  Sciences  and  Technology 
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"But  'glory'  doesn't  mean  'a  nice  knockdown 
argument,'"  Alice  objected. 

"When  I_  use  a  word,"  Humpty  Dumpty  said,  in 
rather  a  scornful  tone,  "it  means  just  what  I 
choose  it  to  mean--neither  more  or  less." 

"The  question  is,"  said  Alice,  "whether  you 
can  make  words  mean  so  many  different  things." 

"The  question  is,"  said  Humpty  Dumpty,  "which 
is  to  be  master--that's  all." 

Through  the  Looking-Glass 
Lewis  Carroll 


"Everybody  talks  about  documentation,  but 
nobody  does  anything  about  it." 

Anonymous 


"You  get  what  you  pay  for." 

Gabriel  Biel 
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I.    Pro1ogue--A  Short  Play:    "Is  Documentation  Documentation?"  or  "Quo 


Vadis  Documentus?" 


Time;    The  present. 

Scene:  The  office  of  a  government  analyst  responsible  for  assessing 
the  Federal  Government's  use  of  policy  evaluation  models.    The  analyst 
is  seated  behind  a  desk,  a  computer  CRT  is  to  one  side  facing  the 
audience.    The  desk  is  piled  with  papers  and  reports  that  describe, 
among  others,  two  of  the  most  important  policy  models  in  current  use — 
the  Transfer  Income  Model  (TRIM)  for  welfare  program  analysis  and  the 
Project  Independence  Evaluation  System  (PIES)  for  energy  policy  analysis 
(34)  and  (40).    Included    amidst  the  material  that  describe  these  two 
models  are  a  user  guide  of  700  plus  typewritten,  double-spaced  pages 
of  text,  tables,  charts  and  programs  for  TRIM  (33);  and  a  set  of  15 
volumes  that  forms  the  latest  PIES  documentation  (44).    As  the  curtain 
rises  the  analyst,  who  appears  confused  and  bleary-eyed,  reaches  out 
and  picks  up  a  paper  at  random  and  begins  to  read. 

Analyst:    "Everyone  agrees  that  documentation  is  a  'good  thing,'..." 
(The  analyst  pauses,  smiles  knowingly,  and  starts  the  next  sentence.) 
"Several  opportunities  are  afforded  when  the  concept  of  documentation 
is  expanded  beyond  a  narrow  technical  description  of  a  piece  of  soft- 
ware...."   (He  breaks  off  and  stares  at  the  two  piles  of  TRIM  and  PIES 
documentation.    His  eyes  wander  down  the  page  and  he  continues  to  read.) 
"Documentation  is  not  fun.    It  is  hard,  nasty  and  boring  business. ..," (37) . 
I    (The  analyst  stops,  puts  down  the  paper  and  reaches  into  the  desk  drawer 
I    and  takes  out  a  package  of  matches.    He  stands,  strikes  a  match  and 
proceeds  to  ignite  the  papers  on  his  desk.    With  a  look  of  relief,  but 
with  a  sense  of  failure,  he  walks  out  the  door,  while  flames  consume 
the  desk.    As  he  exits,  a  moon-shaped,  keep  smiling  face  appears  on  the 
I    CRT  followed  by  an  unending  stream  of  text.) 

Curtain. 


II.  The  Why  of  Model  Documentation--Ref lections  After  Reading  Fifty-plus 
Documents  on  Documentation 

All  general  surveys  of  computer  models  and  all  assessments  of  specific 
models  that  have  come  to  our  attention  conclude  that  the  documents 
that  are  supposed  to  describe  and  explain  these  models  are  either  non- 
existent or  are  lacking  in  detail  and  do  not  serve  the  models  well  (34), 
(35),  (36),  (40),  (45),  (46)  and  (47).    As  will  be  reviewed  later,  many 
proposals  have  been  put  forth  for  improving  and  increasing  the  informa- 
*;!  tion  content  that  describes  a  computer  model.    In  general,  we  agree 
I  with  most  of  these  proposals,  but  we  call  attention  to  our  prologue 
I  to  emphasize  the  need  for  documentation  procedures  that  not  only 
produce  information,  but  do  so  in  a  detailed,  yet  comprehensible, 
clear  and  timely  manner.    This  might  be  an  impossible  task,  especially 
for  such  complex  models  as  TRIM  and  PIES.    (The  prologue  was  not  meant 
to  single  out  the  TRIM  and  PIES  documentation.    We  used  them  to  illustrate 


our  current  abilities  and  difficulties  in  transferring  information  about 
complex,  ongoing  modeling  efforts.) 


Policy  makers  and  other  users  must  understand  these  models.    If  not, 
they  will  either  ignore  the  power  of  models  and  make  decisions  with 
degraded  information,  or  become  blindly  beholden  to  the  computer  output 
and  thus,  relegate  their  decision  responsibilities  to  the  modeling 
analyst.    It  is  the  professional  duty  of  a  modeling  analyst  to  prepare 
documents  that  meet,  in  a  timely  fashion,  the  varied  needs  of  the  sponsor. 
It  is  likewise  the  duty  of  the  model  sponsor  to  state  the  needs  for 
documentation  and  to  provide  personnel,  resources  and  time  to  produce 


Most  manuals  and  reports  that  describe  documentation  requirements  expound 
variations  of  the  same  set  of  purposes  of  documentation  (6),  (8),  (9), 
(25)  and  (35).    Although  these  reports  mainly  describe  the  documentation 
issues  of  computer  software  and  not  computer  models,  per  se,  we  feel 
that  they  also  address  model  documentation  concerns.    A  composite  listing 
of  documentation  purposes  for  models  and  software  includes  the  following: 

•  to  record  technical  information  that  enables  system 
and  program  changes  to  be  made  quickly  and  effectively 

•  to  enable  programmers  and  systems  analysts  other  than 
originators  to  use  and  to  work  on  the  programs. 

•  to  assist  the  user  in  understanding  what  has  been  done. 

§     to  increase  the  model  and  program-sharing  potential. 

0     to  facilitate  auditing  and  verification  of  the  model  and 
the  program  operations,  i.e.  model  evaluation. 

f     to  provide  managers  with  information  to  review  at 

significant  developmental  milestones  to  determine  that 
requirements  have  been  met  and  that  resources  should 
continue  to  be  expended. 

•  to  reduce  the  disruptive  effects  of  personnel  turnover. 

f     to  facilitate  understanding  among  managers,  developers, 
programmers,  operators,  and  users  by  providing  information 
about  maintenance,  training,  changes  and  operation  of  the 
software  (model). 

0     to  inform  other  potential  users  of  the  functions  and 
capabilities  of  the  software  (model),  so  that  they  can 
determine  whether  it  will  serve  their  needs. 


]_/  Our  comments  here  and  throughout  are  not  limited  to  policy  evaluation 
models,  but  apply  to  all  complex  computer  models. 
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Many  claims  are  made  as  to  the  importance  of  documentation  of  computer 
software  systems:  "To  maximize  the  return  on  this  investment  (in 
computer  systems),  and  to  provide  for  cost-effective  operation, 
revision,  and  maintenance,  sufficient  documentation  is  needed  at 
each  stage  of  the  software  development  life  cycle,"  (8);  and  "Documentation 
provides  the  means  for  the  greatest  and  most  efficient  utilization  of 
the  system  by  the  user. ..and  the  means  for  careful,  well-planned 
design  and  integration  of  the  system,"  (50). 

If  the  purposes  and  claims  for  documentation  are  true,  then  the  modeling 
and  computer  programming  communities  appear  to  be  grossly  derelict  in 
their  duties  in  that  the  "lack"  of  documentation  is  one  of  the  main 
reasons  cited  for  model  failures,  or  for  models  being  little  used  by 
their  sponsors,  or  for  models  not  being  usable  by  others  (35),  (45), 
and  (46).    A  recent  book  on  large-scale  models  begins  a  chapter  on 
documentation  by  lamenting  "Documentation  is  one  of  the  most  neglected 
aspects  of  modeling  and  simulation,  partly  because  it  is  largely 
non-creative  and  therefore  uninteresting.    Furthermore,  because  it 
should  be  everyone's  responsibility,  it  frequently  becomes  the  responsi- 
bility of  no  one.    While  documentation  should  commence  at  the  very 
beginning  of  a  project,  it  is  often  left  until  the  project  is  otherwise 
complete.    This  in  turn  makes  documentation  more  difficult  because 
it  requires  searching  old  records.    In  addition,  most  workers  find 
documentation  distasteful  because  it  is  part  of  the  cleanup  operation,"  (41 ) . 


How  can  we  explain  the  difference  between  what  we  expect  of  documentation 
and  what  we  actually  get?    If  a  model  for  documentation  was  constructed 
using  the  set  of  purposes  and  objectives  of  documentation  as  assumptions, 
it  could  not  be  validated  using  the  results  of  most  real-world  documen- 
tation.   We  can  only  conclude  that  the  general  run  of  sponsors 
and  users,  model  development  managers,  analysts  and  programmers  do  not 
believe  in  the  high- tone  purposes  of  documentation;  or  that  the  normal 
or  usual  circumstances  of  model  development  precludes  the  production 
of  documentation  that  meets  the  varied  needs  of  a  model's  audience.  Of 
course,  a  combination  of  these  two  conclusions  can  hold,  along  with 
variations,  e.g.,  a  model  was  originally  a  research  tool  and  was 
subsequently  elevated  to  forming  the  core  of  a  policy  evaluation 
model . 

Information  that  describes  the  current  state  of  models  causes  us  to  be 
pessimistic  about  modeling  activities  with  respect  to  modeling  documen- 
tation.   In  the  survey  (46)  it  is  noted  that  documentation  was  considered 
inadequate  to  enable  other  project  personnel  to  set  up  and  run  the  models 
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(surveyed)  in  about  75  percent  of  the  cases.    To  this  we  may  respond  with 
the  question:  "Was  the  transferability  of  the  model  an  objective  of  the 
project  and  if  so,  was  it  known  to  the  project  personnel  and  conveyed  to 
them  by  the  allocation  of  resources  and  time?"    The  first  part  of  the 
question  was  probably  true  if  for  no  other  reason  than  that  most  models 
for  the  Federal  Government  are  developed  externally  by  contractors  and 
grantees,  and  everyone  knows  that  analysts  and  programmers  move  about 
very  rapidly  from  job  to  job,  if  not  to  different  organizations.  Whether 
proper  resources  were  committed  is  the  real  unknown;  and  if  so,  were  the 
resources  used  properly? 

How  to  improve  the  documentation  activities  for  computer  models  and 
whether  such  improvement  would  really  overcome  any,  let  alone  all  of 
the  problems  attributed  to  the  lack  of  documentation,  is  an  open  question. 
Does  the  answer  lie  with  the  lack  of  motivation  in  modeling  personnel  and 
how  they  really  view  the  purposes  of  documentation  and/or  the  emphasis 
placed  on  documentation  by  the  sponsors/users?    We  need  to  remember  that 
the  economic  cost  of  producing  an  item  like  model  documentation  must  be 
compared  to  its  intrinsic  and  real  value.    One  wonders  whether  the  cost 
in  dollars  and  modeling  labor  (applied  to  unglamorous  documentation)  is 
so  great  that  the  modeling  world  is  willing  to  live  with  (and  lament) 
the  inefficiencies  caused  by  not  producing  worthwhile  and  useful  docu- 
mentation. 

In  the  following  sections,  we  shall  review  a  wide  range  of  computer 
model  documentation  proposals,  issues  and  topics  in  the  hope  that  the 
sharing  of  the  collective  knowledge  of  many  investigators'  will  offer 
some  directions  for  research  and  operational  approaches  to  resolving 
this  open  question. 


Ill    The  What  of  Model  Documentation--Further  Reflections 

Of  the  material  on  documentation  that  has  come  to  our  attention,— ^most 
of  it  describes  documentation  standards  or  guidelines  designed  for 
automated  data  systems  (ADS).    It  is  our  view  that  documentation  require- 
ments for  ADS  form  a  subset  of  the  documentation  needs  for  computer 
models,  and  that  we  can  build  upon  the  ADS  experiences  to  benefit  model 
documentation.    However,  a  complex  policy  or  decision  model  has  infor- 
mation and  documentation  needs  beyond  that  of  a  complex  software  system. 
We  shall  review  some  of  the  ADS  documentation  structures  and  modify  and 
combine  them  with  model  documentation  proposals  to  obtain  what  we  feel 
are  documentation  requirements  and  an  approach  to  documentation  for 
computer  models. 


V  Requests  for  documentation  guidelines  and  standards  for  computerized 
models  and  documentation  examples  were  sent  to  over  100  individuals 
and  organizations.    Their  responses  are  deeply  appreciated  and  grate- 
fully acknowledged.    The  applicable  reports  are  cited  in  the 
References. 
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First,  what  is  model  documentation?  From  (34j  and  (40)  we  have  that 
computer  documentation  is  defined  as  information  recorded  during  the 
design,  development,  and  maintenance  of  computer  applications  to  explain 
pertinent  aspects  of  a  data  processing  system;  including  purposes, 
methods,  logic,  relationships,  capabilities,  and  limitations.  Further, 
computer  model  documentation  is  the  principal  instrument  which  allows 
people  in  a  modeling  effort--the  user,  the  model  developer,  potential 
users,  etc. --to  communicate,    complete  documentation  Is  important  to 
(1 j  ensure  that  the  model  is  thoroughly  understood  and  can  be  operated 
and  maintained  in  the  present  and  the  future,  and  {Z)  facilitate  eval- 
uation of  the  model  by  a  third  party  (i.e.,  someone  other  than  the 
model  developer  or  initial  user).    The  adequacy  of  model  documentation 
depends  on  the  answer  to  the  question  "Is  the  computer  model  documen- 
tation sufficient  to  understand,  use  and  maintain  the  model?" 

The  above  definitions  and  question  automatically  pose  a  number  of 
related  questions  that  have  been  addressed  by  model  documentation 
researchers,  but  have  yet  to  be  answered  with  confidence;  possibly  they 
cannot.    Some  of  these  questions  are:    Exactly  what  type  of  information 
is  required  to  form  complete  and  adequate  computer  model  documentation? 
How  and  when  should  it  be  made  available?    Who  does  tne  writing  and  how 
should  they  be  motivated?    How  are  changes  made  to  ensure  currency  of 
documentation?    How  does  the  extent  of  documentation  vary  with  the 
model's  use,  complexity,  cost,  etc.?    What  is  the  process  by  which 
adequate  documentation  can  be  obtained?    We  will,  in  this  report,  address 
these  and  related  issues  by  summarizing  the  views  of  others  and  by 
offering  our  thoughts  that  have  been  formed  by  past  experiences  and  the 
present  study. 

A  basic  but  reasonable  view  of  what  elements  of  information  should  be 
included  in  a  model's  documentation  would,  we  feel,  converge  to  the 
following  (35): 

•  A  precise  statement  of  what  the  model  is  supposed  to  do. 

•  The  mathematical /logical  definition,  assumptions,  and 
formulation  of  the  problem  being  modeled. 

•  A  complete  set  of  current  input  and  output  and  test  cases 
that  have  been  run. 

•  A  complete  set  of  flow  charts  of  the  computer  program. 

•  A  set  of  operating  instructions  for  the  computer  operator. 

0      An  explanation  of  the  various  options  available  in  using 
the  model . 

•  Ihe  computer  program  itself  (listing),  with  comments  about 
various  operations  in  the  program. 
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•      The  names  of  the  programmers  and  project  managers 
responsible  for  the  model  development  and  computer 
program. 

The  above  elements  are  open  to  interpretation  and  a  project  staff  could 
satisfy  their  explicit  meanings  without  producing  adequate  documentation. 
This  would  be  especially  true  if  a  complex  policy  model  were  under  con- 
sideration.   What  we  require  is  a  process  for  obtaining  specific, 
detailed  information,  organized  to  satisfy  the  needs  of  each  segment  of 
a  model's  audience.    We  shall  describe  such  a  process  subsequently. 


IV.    Guidelines  for  Computer-Program  Documentation  and  Their 
Relationship  to  Model  Documentation 

In  this  section,  we  review  a  number  of  reports  that  are  designed  to 
guide  and  direct  analysts  and  computer  programmers  in  preparing  docu- 
mentation of  computer  software.    Where  appropriate,  we  emphasize  those 
aspects  that  relate  to  the  documentation  of  computer  models. 

Historically,  one  of  the  first  formal  and  official  set  of  Government 
ADS  standards  was  published  in  December  ly72  by  the  Department  of 
Defense  (DoD).L'  Although  the  use  of  this  manual  (b)  is  mandatory,  a 
realistic  philosophy  as  to  its  applicability  is  conveyed.    It  is  noted, 
for  example,  that  documentation  standards  deal  with  the  communication 
of  information  that  cannot  be  rigidly  standardized  and  thus,  modifica- 
tions by  a  project  manager  are  allowed.    The  manual  stresses  (possibly 
unrealistically)  that,  although  the  process  of  documentation  is  often 
identified  as  a  separate  phase  of  project  development,  documentation 
activity  must  be  a  continuing  part  of  the  developmental  effort  and  not 
relegated  and  delayed  to  the  last  stages  of  a  project. 

The  DoD  ADS  documentation  manual  (6j  among  others,  formed  the  basis  for 
the  important  and  more  generally  applicable  Federal  ADS  guidel ine--the 
NBS  issued  Federal  Information  Processing  Standards  Publication  38 
(FIPS  PUB  38)  titled  "Guidelines  for  Documentation  of  Computer  Programs 
and  Automated  Data  Systems,"  (8).    These  guidelines  are  intended  to  be  a 
basic  reference  and  checklist  for  planning  and  evaluation  of  documenta- 
tion throughout  the  Federal  Government.    In  the  discussion  that  follows, 
we  shall  describe  elements  of  (6)  and  [S)  that  we  feel  are  of  value  to 
the  aims  of  computer  model  documentation. 

The  key  to  the  DoD  and  NBS  documentation  guidelines  is  the  definition  of 
ten  major  document  types  and  the  relating  of  the  production  of  these 
documents  to  the  phases  and  stages  of  the  software. life  cycle.  This 
life  cycle  is  defined  by  the  following  major  phases  ^8): 

•       Initiation  Phase:    During  this  phase  the  objectives  and 
general  definition  of  the  requirements  for  the  software 


1/  An  earlier  set  of  documentation  guidelines  was  published  by  NASA  (b2) 
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are  established.    Feasibility  studies,  cost-benefit 
analyses,  and  the  documentation  prepared  within  this 
phase  are  determined  by  agency  procedures  and 
practices. 

•  Development  Phase:    During  this  phase,  the  requirements 
for  the  software  are  determined  and  the  software  is  then 
defined,  specified,  programmed  and  tested.    The  ten  major 
documents  are  prepared  in  this  phase  to  provide  an 
adequate  record  of  the  technical  information  developed. 

•  Operation  Phase:    During  this  phase,  the  software  is 
maintained,  evaluated  and  changed  as  additional  require- 
ments are  identified.    The  documentation  is  maintained 
and  updated  accordingly. 

The  development  phase  of  the  software  life  cycle  is  subdivided  into  four 
main  stages  as  follows: 

Definition  Stage:    When  the  requirements  for  the  software  and 
documentation  are  determined. 

Design  Stage:    When  the  design  alternatives,  specific 
requirements,  and  functions  to  be  performed  are  analyzed 
and  a  design  is  specified. 

Programming  Stage:  When  the  software  is  coded  and 
debugged. 

Test  Stage;    When  the  software  is  tested  and  related 
documentation  is  reviewed.    The  software  and  documen- 
tation are  evaluated  in  terms  of  readiness  for  imple- 
mentation. 

The  ten  document  types  that  are  to  be  produced  during  the  development 
phase  are  summarized  in  Figure  1.    The  relationship  of  these  documents 
to  the  stage  in  which  they  may  be  produced  is  shown  in  Figure  2. 
FIPS  PUB  38  notes  that  the  terminology  used  to  desct^ibe  the  stages  is 
arbitrary,  but  that  it  provides  a  convenient  framework  within  which  the 
development  of  the  ten  document  types  may  be  discussed.    It  is  also 
stressed  that  not  all  document  types  are  required  to  document  software 
in  every  case,  and  that  in  some  cases  the  various  document  types  may 
need  to  be  combined. 

We  shall  not  discuss  the  specific  contents  of  the  ten  documents  as  that 
is  the  function  of  (6)  and  (8);  detailed  content  outlines  are  given  in 
those  publications.  However,  what  is  of  interest  are  the  philosophy  and 
approach  to  some  of  the  general  problems  and  considerati-ons  of  documenta- 
tion discussed  in  (6)  and  (8),  and  related  publications  such  as  (25). 
We  review  some  of  these  aspects  next, 

A  documentation  plan  must  be  developed  early  in  the  project.  Under 
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Functional  Requirements  Document.  The  purpose  of  the  Functional  Requirements 
Document  is  to  provide  a  basis  for  the  mutual  understandinsr  between  users  and  designers 
of  the  initial  definition  of  the  software,  including  the  requirements,  operating  environment, 
and  development  plan. 

Data  Requirements  Document.  The  purpose  of  the  Data  Requirements  Document  is 
to  provide,  during  the  definition  stage  of  software  development,  a  data  description  and  tech- 
nical information  about  data  collection  requirements. 

System/Subsystem  Specification.  The  purpose  of  the  System/Subsystem  Specifica- 
tion is  to  specify  for  analysts  and  programmers  the  requirements,  operating  environment, 
design  characteristics,  and  program  specifications  (if  desired)  for  a  system  or  subsystem. 

Program  Specification.  The  purpose  of  the  Program  Specification  is  to  specify  for 
progi'ammers  the  requirements,  operating  environment,  and  design  characteristics  of  a  com- 
puter program. 

Data  Base  Specification.  The  purpose  of  the  Data  Base  Specification  is  to  specify 
the  identification,  logical  characteristics,  and  physical  characteristics  of  a  particular  data  base. 

Users  Manual.  The  purpose  of  the  Users  Manual  is  to  sufficiently  describe  the  func- 
tions performed  by  the  software  in  non-ADP  terminology,  such  that  the  user  organization  can 
determine  its  applicability  and  when  and  how  to  use  it.  It  should  serve  as  a  reference  docu- 
ment for  preparation  of  input  data  and  parameters  and  for  interpretation  of  results. 

Operations  Manual.  The  purpose  of  the  Operations  Manual  is  to  provide  computer 
operation  personnel  with  a  description  of  the  software  and  of  the  operational  environment 
so  that  the  software  can  be  run. 

Program  Maintenance  Manual.  The  purpose  of  the  Program  Maintenance  Manual  is 
to  provide  the  maintenance  programmer  with  the  information  necessary  to  understand  the 
programs,  their  operating  environment,  and  their  maintenance  procedures. 

Test  Plan.  The  purpose  of  the  Test  Plan  is  to  provide  a  plan  for  the  testing  of 
software;  detailed  specifications,  descriptions,  and  procedures  for  all  tests;  and  test  data  re- 
duction and  evaluation  criteria. 

Test  Analysis  Report.  The  purpose  of  the  Test  Analysis  Report  is  to  document  the 
test  analysis  results  and  findings,  present  the  demonstrated  capabilities  and  deficiencies  for 
review,  and  provide  a  basis  for  preparing  a  statement  of  software  readiness  for  implementa- 
tion. 


Software  Life  Cycle  Docuraent  Types  [8] 
Figure  1 
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guidance  from  the  agency,  the  project  manager  needs  to  determine  a  plan 
that  details: 

•  What  document  types  apply' and  should  be  prepared. 

•  The  formality,  extent  and  detail  of  the  documentation. 

•  Procedures  and  schedule  of  review,  approval  and 
distribution,  and  the  distribution  list. 

•  Responsibilities  for  documentation  maintenance  and 
change  control  through  the  development  phase. 

The  formality,  extent  and  level  of  detail  of  the  documentation  is  a 
direct  function  of  the  size,  complexity  and  risk  of  a  project.    FIPS  PUB 
38  offers  the  following  two  schemes,  A  and  B,  for  determining  how  criteria 
could  be  established  to  aid  project  managers  in  stating  the  extent  and 
level  of  detail  of  documentation  required. 

Scheme  A 

Four  levels  of  documentation  are  defined: 

Level  1  -  Minimal  level  guidelines  are  applicable  to  single  use  programs, 
one-shot  jobs  of  minimal  complexity. 

Level  2  -  Internal  level  guidelines  apply  to  special  purpose  programs 
which  appear  to  have  no  sharing  potential  and  are  for  use  only  by  the 
requesting  user;  also  to  large  programs  which  have  a  short  life  expec- 
tancy. 

Level  3  -  Working  document  level  applies  to  programs  which  are  expected 
to  be  used  by  a  number  of  people  in  the  same  installation  or  which  may  be 
transmitted  on  request  to  other  installations  or  to  contractors  or 
grantees. 

Level  4  -  Formal  publication  level  applies  to  programs  which  are  of 
sufficient  general  interest  and  value  to  be  announced  outside  the  origin- 
ating installation;  also  included  are  those  programs  which  are  critical 
to  the  activities  of  the  installation. 

Further  definition  of  these  levels  in  terms  of  use,  project  cost  and 
resultant  documentation  requirements  are  summarized  in  Figure  3. 

Scheme  B 

This  procedure  employs  twelve  criteria,  with  weighting  factors,  and  a 
scale  of  the  total  weighted  criteria  to  establish  formal  documentation 
requirements.    Table  1  shows  the  weighted  criteria,  and  Table  2  illus- 
trates their  application.    The  way  to  use  these  tables  is  to: 
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Level 

If  PROJECT  COST: 

Or  USAGE 

Then  DOCUMENTATION 
ELEMENTS 

And  EXTENT  OF  EFFORT 

1 

Less  than  $10Uo 
Or 

One  Man-month 

One  Shot 
(Single  Use) 

Software.  Summary  plus  any 
incidentally  produced  docu- 
mentation. 

No  special  effort,  normal  good  prac- 
tice. 

2 

$1000  to  $5000 

Special  or 
Limited 
Purpose  or 
Application 

Level  1  plus  Users  Manual 
and  Operations  Manual. 

Minimal  documentation  effort,  spent 
on  informal  documentation.  No  for- 
mal documentation  effort. 

3 

Over  $5000 

Multipurposed, 
or  Multiuser 

Level  2  plus  Functional  Re- 
quirements Document,  Pro- 
gram Specification,  Pro- 
gram Maintenance  Manual, 
Tp<?t    Plan     Tpst  Anfllv<?is 

•iLCOv       A   lolly        ACOb             114X1  olO 

Report,  and  System/Sub- 
system Specification. 

All  basic  elements  of  documentation 
should  be  typewritten,  but  need  not 
be  prepared  in  finished  format  for 
publication  or  require  external  edit 

4 

Over  $5000 

Publicly 

Announced,  or 
Critical  to 
Operations 

Level  3  produced  in  a  form 
suitable  for  publication. 

At  a  minimum,  all  basic  elements  pre- 
pared for  formal  publication,  in- 
cluding external  review  and  edit. 

Cost  and/or  Usage  Threshold  Criteria- -Scheme  A  [8] 

Figure  3 
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WEIGHTS 

Criteria 

1 

2 

3 

4 

5 

1. 

Originality 
required 

None — reprogram 
on  different 
equipment 

Minimum — more 
stringent 
requirements 

Limited — new 
interfaces 

Considerable —  apply 
existing  state  of  art 
to  environment 

Extensive — requires 
advance  in  state  of 
the  art 

2. 

Degree  of 
generality 

Highly  restricted. 
Single  purpose 

Restricted —  parameter- 
ized for  a  range  of 
capacities 

Limited  flexibility. 
Allows  some  change 
in  format 

Multi-purpose.  Flexible 
format.  Range  of 
subjects 

Very  flexible — able  to 
handle  a  broad 
range  of  subject 
matter  on  different 
equipment 

3. 

Span  of  operation 

Local  or  utility 

Component  command 

Single  command 

Multi-command 

Defense  Department. 
World  wide. 

4. 

Change  in  scope 
and  objective 

None 

Infrequent 

Occasional 

Frequent 

Continuous. 

5. 

Equipment 
complexity 

Single  machine. 
Routine 
piocessing 

Single  machine.  Routine 
processing.  Extended 
peripheral  system 

Multi-computer. 
Standard  peripheral 
system 

Multi-computer.  Ad- 
vanced programming. 
Complex  periphera) 
system 

Master  control  system. 
Multi-computer  auto 
input/output  and 
display  equipment. 

6. 

Personnel 

assigned 

1—2 

3—5 

5—10 

10—18 

18  and  over 

7. 

Developmental 
cost 

1-lOk 

10-60k 

50-200k 

200-600k 

Over  500k 

8. 

Criticality 

Data  processing 

Routine  operations 

Personnel  safety 

Unit  survival 

National  defense 

9. 

Average  response 
time  to  program 
change 

2  or  more  weeks 

1—2  weeks 

3—7  days 

1—3  days 

1-  24  hours 

10. 

Average  response 
time  to  data 
inputs 

2  or  more  weeks 

1-2  weeks 

1-7  days 

1-24  hours 

0—60  minutes 

11. 

Programming 
languages 

High  level 
language 

High  level  and  limited 
assembly  language 

High  level  and  ex- 
tensive assembly 
language 

Assembly  language 

Machine  language 

12. 

Concurrent 
software 
devclopnMnt 

None 

Limited 

Moderate 

Extensive 

Exhaustive 

Weighting  Procedure  for  Twelve  Documentation  Criteria- -Scheme  B  [8] 

Table  1 
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TOTAL 
WEIGHTED 
CRITERIA 

Software 
Summary 

Users 
Manual 

Operations 
Manual 

Program  Maintenance 

Manual 

Test 
Plan 

Functional  Require- 
ments Document 

System/Subsystem 
Specification 

Test  Analysis 
Report 

Program 
Specification 

Data  Requirements 
Document 

Data  Base 
Specification 

A  1  O  41 
0-l<4 

X 

12-15 ' 

X 

X 

i  O  tin 

X 

X 

X 

X 

X 

** 

*** 

**« 

24-38 

X 

X 

X 

X 

X 

X 

*** 

36-50 

X 

X 

X 

X 

X 

X 

X 

X 

*** 

48-60 

X 

X 

X 

X 

X 

X 

X 

X 

X 

*** 

NOTES:       *  Additional  document  types  may  be  required  at  lower  weighted  criteria  totals  to  satisfy  local 
requirements. 

**  The  Test  Analysis  Report  logically  should  be  prepared,  but  may  be  informal. 

Preparation  of  the  Data  Requirements  Document  and  Data  Base  Specification  is  situationally 
dependent. 


Total  Weighted  Documentation  Criteria 
vs.  Required  Documentation  Types- -Scheme  B  [8] 

Table  2 
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weight  the  software  by  each  of  the  twelve  criteria. 
Table  1. 

Sum  the  weights  assigned  to  obtain  the  total  weighted 
criteria. 

For  the  total  weight,  find  the  row  in  Table  2  that  lists 
the  documents  to  be  prepared. 

The  ADS  documentation  process  described  above  (and  in  detail  in  (6)  and 
(8))  should  be  viewed  as  basic  and  central  to  the  documentation  require- 
ments of  complex  computer  models.    However,  we  do  not  think  that  it  cap- 
tures the  total  information  needs  of  a  modeling  project — especially  of 
a  policy  or  decision  model--and  it  must  be  expanded.    We  will  do  this  in 
Section  VI  by  relating  model  documentation  to  the  concept  of  a  (to  be 
defined)  model  life  cycle.    We  next  continue  our  review  of  other  ADS 
documentation  procedures  for  their  insights  into  the  general  problem  of 
computer  program  documentation. 

It  is  interesting  to  note  that  even  with  the  DoD  (6)  and  FTPS  PUB  38  (8) 
guidelines.  Federal  agencies  are  still  developing  and  using  different 
computer  program  documentation  procedures  (10),  (11),  (27),  and  (54). 
This  emphasizes  that  documentation  is  often  viewed  as  a  local  concern. 
However,  the  flexibility  of  (6)  and  (8)  allows  for  local  interpretations, 
while  still  affording  a  framework  for  proper  documentation  decisions. 
For  example,  in  (10)  and  (11)  we  find  a  listing  of  computer  program 
documentation  contents  that  are  narrower  and  more  limited  than  the  FIPS 
PUB  38  requirements.    Although  we  can  envision  situations  that  call  for 
local  documentation  guidelines,  it  is  unclear  why  (6)  and  (8)  could  not 
be  adapted  to  these  local  needs,  or  at  least  be  used  to  form  the  core 
of  any  in-house  documentation  procedure.    Special  local  guidelines  might 
be  very  adequate  and  appropriate,  but  if  documentation  is  ever  to  be 
more  than  an  afterthought  of  computer  models  and  systems,  then  the  anal- 
ysts and  programmers  must  be  made  aware  of  and  taught  to  use  accepted 
(if  not  universal)  documentation  practices. 

Appropriate  to  local  control  and  development  are  the  guidelines  that 
ensure  consistency  in  computer  programs  written  at  a  certain  installation 
and/or  for  a  specific  purpose,  e.g.,  (11).    These  type  of  standards 
are  for  subroutine  linkage,  FORTRAN  coding  conventions,  etc.,  and  are 
found  to  be  a  part  of  the  operating  environments  of  successful  computer 
installations.    The  programming  guidelines  (11)  address  documentation  in 
an  interesting  manner.    First,  it  realistically  notes  that  there  are 
usually  two  problems  regarding  computer  program  documentation:  (1)  it 
does  not  exist,  or  (2)  it  exists  but  is  out-of-date.    The  recommended 
solution  to  both  these  problems  is  concurrency,  i.e.,  documentation 
must  be  written  concurrently  with  program  development  in  order  for  it  to 
be  accurate  and  available  when  checkout  is  complete. 

In  (11),  two  types  of  documentation  are  differentiated:    (1)  inline- 
documentation  which  appears  in  the  program  code  itself  and  useful  to 
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someone  attempting  to  understand  and  maintain  the  program,  and 
(2)  formal— documentation  that  includes  all  reports  necessary  to  define 
the  model  adequately.    This  particular  installation  uses  a  program  design 
language  (PDL)  (53)  as  a  tool  in  defining  and  documenting  the  design,  and 
a  specially  developed  computer  assisted  documentation  (CAD)  technique  (12). 
Both  such  elements  are  quite  appropriate  for  computer  model  development 
and  documentation. 

i  Another  documentation  element  of  local  concern  is  the  general  makeup  of 
the  documents.    For  example,  the  report  (7)  specifies  the  physical 
appearance,  form  and  components  of  technical  computer  program  and  system 
documentation.    It  is  to  be  applied  to  the  ten  FIPS  PUB  38  documents  and 
it  details  page  layouts.,  flow  chart,  graphic  symbols,  etc.    Many  agencies 
have  similar  documents,  e.g.,  (19). 

I 

'The  Army  regulation  (13)  describes  in  great  detail  the  objectives,  poli- 
icies  and  procedures  and  assigns  responsibilities  for  the  formulation, 
"design,  development,  testing,  evaluation,  installation,  operation,  main- 
j tenance  and  review  of  Army  Management  Information  Systems  (AMIS).  Of 
i interest  is  the  classification  of  AMIS  by  multi-users  and  man-years  of 
I  effort.    A  Class  A  is  a  multi-user  system  and/or  one  that  requires  more 
I  than  fifteen  man-years;  a  Class  B  is  a  single-user  system  requiring 
I  between  three  and  fifteen  man-years  of  effort;  while  a  Class  C  is  a 
single-user  system  which  requires  less  than  three  man-years  of  effort. 
Each  such  class  requires  a  minimum  level  of  documentation  that  is  con- 
tained in  only  four  types  of  manuals:  systems  description,  functional 
I  user,  program  description  and  operator's  manual.    It  is  not  clear  why 
I  the  DoD  ADS  (6)  or  the  FIPS  PUB  38  (8)  are  not  explicitly  cited,  even 
though  it  is  stated  in  (13)  that  FIPS  (in  general)  must  be  incorporated 
in  all  new  systems  design  or  major  system  change  efforts. 

The  report  (18)  reviews  all  Air  Force  software  documentation  requirements. 
It  identifies  major  standard  Air  Force  data  items  which  apply  to  software, 
its  development,  acquisition  and  use.    The  term  "data  item"  is  used  to 
refer  to  a  formal  collection  of  information  (data)  acquired  during  the 
system  acquisition  process  to  support  the  management  of  technical  objec- 
tives of  the  program.    These  are  termed  Data  Item  Descriptions  (DID)  and 
the  report  lists  sixty-five  DIDs  to  software  acquisition,  development, 
;  maintenance  and  use;  with  twenty-five  designated  as  major  software  docu- 
iments.    Each  of  the  latter  meets  one  of  the  following  criteria: 

It  is  unique  to  software  and  this  uniqueness  is  significant 
in  proper  use  of  the  document. 

It  is  not  unique  to  software,  but  it  can  provide  visibility 
in  critical  areas  of  software  development. 

'Many  of  these  major  documents  are  similar  to  those  of  (6)  and  (8),  with 
some  special  reports  identified,  e.g.,  human  operator  critical  task 
analysis.    A  minimum  set  of  software  documents  is  also  identified; 
twelve  documents  from  the  major  list  and  five  related  documents. 
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The  twenty-five  major  documents  are  described  in  (18)  in  terms  of: 


purpose  of  the  document. 

references  to  more  detailed  descriptions  of  content 
and  organization. 

who  originates  the  document. 

categories  of  usage  in  the  phases  of  the  computer 


special  conditions  necessary  for,  or  requiring  th« 
application  of  the  document. 

relationships  to  other  documents, 

adequacy  of  document  delivered. 

potential  problems  in  using  the  document. 


The  report  (18)  also  compares  its  documentation  requirements  with  the 
DoD  ADS  documentation  standards  given  in  (6).    Those  of  (18),  which  are 
more  extensive  and  inclusive,  are  designed  to  apply  to  the  acquisition 
of  a  large  software  system  composed  of  software,  personnel,  equipment 
and  communications  subsystems  (as  in  a  command  and  control  system),  in 
contrast  to  an  ADS  oriented  to  software  only.    Software  in  (18)  is 
defined  as  computer  programs  and  computer  data,  with  the  former  also 
including  applications  and,  by  association,  computer  models. 

An  early  handbook  for  computer  systems  documentation  is  the  AEC  specifi- 
cations (19).    Here,  six  major  documents  are  defined  and  divided  into 
two  sets:  (1)  historical  documents--prel iminary  study  report,  perfor- 
mance and  design  specifications,  programming  specifications;  (2)  oper- 
ational documents--system  manual,  program  manual,  user's  manual.  The 
latter  set  is  the  one  kept  current  by  updating  as  changes  occur  to  the 
system,  while  the  former  set  is  of  historical  value  once  each  report 
has  served  its  purpose.    Most,  if  not  all  of  information  required  by 
these  six  reports  is  also  required  by  the  more  inclusive  ten  FIPS  PUB  38 
reports  (8).    Of  importance  to  computer  model  documentation  concerns  is 
that  (19)  calls  for  a  separate  preliminary  study  report  (basically  a 
problem  definition  with  a  recommended  course  of  action,  supported  by 
cost  and  other  data),  while  FIPS  PUB  38  includes  similar  information 
into  a  broader  functional  requirements  document  generated  during  the 
software  definition  stage.    It  is  felt  that  for  a  computer  model  such 
preliminary  information  should  be  highlighted  to  stress  user  interest 
and  future  involvement  in  the  results. 


y  The  Air  Force  computer  program  life  cycle  is  segmented  in  to  the 
following  phases:  analysis,  design,  code  and  checkout,  test  and 
implementation,  installation,  and  operation  and  support. 
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The  EPA  has  published  an  Automatic  Data  Processing  (ADP)  Manual  (27) 
that  establishes  policy  responsibilities  and  procedures  for  the  manage- 
ment and  operations  of  the  EPA  ADP  program.    The  manual  utilizes  an  ADP 
system  development  cycle  consisting  of  five  main  steps:  system  feasibility 
study,  system  design  specification,  system  development  and  implementation, 
system  operation  and  maintenance,  and  periodic  review  and  audit.  Although 
not  directed  to  computer  models  per  se,  the  manual  is  to  be  applied  when 
carrying  out  a  program  for  exploitation  of  scientific  and  technical  appli- 
cation of  computers  and  mathematical  and  statistical  approaches  to  the 
needs  of  the  EPA. 

The  manual  (27)  includes  a  chapter  on  documentation  standards  and  require- 
ments that  delineates  the  information  to  be  included  in  the  system  feas- 
ibility study,  system  design  specifications,  user  manual  and  system  main- 
tenance documentation.    Again,  PIPS  PUB  38  (8)  documentation  reports  are 
more  inclusive.   What  is  of  interest  is  the  EPA  requirement  of  a  per- 
iodic review  and  audit  (step  five  of  the  ADP  system  development  cycle 
above).    We  will  discuss  subsequently  this  concept  as  applied  to  model 
documentation.    For  now,  we  note  that  the  EPA  manual  states  that  ADP 
systems  should  be  reviewed  and  audited  on  an  annual  basis  to  determine: 

if  the  system  is  still  needed  to  satisfy  valid  EPA 
requirements, 

if  the  system  is  performing  adequately  or  needs  to  be 
modified, 

if  the  costs  involved  with  operating  the  system  are 
justified  by  the  benefits  received, 

if  the  costs/benefits  associated  with  the  system  justify 
its  continued  existence,  given  the  current  fund  limitations 
and  overall  priorities  within  the  user's  organization,  and 

if  adequate  user  documentation  and  system  maintenance 
documentation  exists  to  use  and  maintain  the  system. 

The  Transportation  Systems  Center  (TSC)  of  the  Department  of  Transpor- 
tation has  issued  guidelines  for  the  documentation  of  computer  programs 
(20).    As  the  documentation  outline  given  in  the  guidelines  forms  a 
concise  statement  of  the  minimum  information  required  to  document  a 
computer  model,  we  present  it  in  Appendix  A. 

The  TSC  guidelines  and  those  of  the  Department  of  Labor  (14)  also  include 
a  set  of  criteria  for  determining  the  level  and  extent  of  the  documenta- 
tion effort.    They  are  similar  to  the  criteria  of  Figure  3. 

Other  documentation  approaches  are  given  in  (16),  (21,),  (22),  (26),  (28), 
(31),  and  (58).    Reports  (16)  and  (28)  describe  the  use  of  notebooks  to 
aid  in  the  documentation  of  both  computer  programs  and  models. 
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The  program  notebook  (16)  is  organized  to  encourage  planning  and  com- 
munication, as  well  as  basic  documentation.    It  contains  a  record  of  on- 
going activity  and  accomplishments  so  that  a  supervisor  or  subsequent 
programmer  can  ascertain  the  status  of  the  task.    In  addition,  it  pro- 
vides source  material  for  any  formal  documentation  that  is  required. 
The  program  notebook  includes  an  event  log  (a  historical  picture  of 
relevant  activities)  and  a  compendium  of  working  documentation  produced 
during  the  phases  of  software  development.    The  notebook  maintains  a 
current  record,  and  material  no  longer  pertinent  is  removed  or  indicated 
as  being  out  of  date. 

The  notebook  of  (28)  is  designed  to  keep  a  well-ordered  and  comprehensive 
record  of  information,  analyses  and  models  employed  in  the  course  of  a 
simulation  study.    It  is  an  attempt  to  capture  the  working  assumptions, 
plans  and  progress  of  the  modeler  as  an  aid  in  improving  the  modeling 
process  and  model  documentation.    Both  types  of  notebooks  are  very 
appropriate  for  any  computer  modeling  project. 

An  example  of  a  well-considered  set  of  documentation  standards  developed 
by  a  non-government  group  is  the  report  (31),  written  by  the  computer 
services  group  of  SRI.    Four  kinds  of  documents  are  discussed:  user's 
guide,  user's  reference  manual,  programmer's  maintenance  manual  and 
subprogram  brief.    The  report  details  in  outline  and  narrative  forms 
material  to  be  included  in  each  document.    It  is  structured  to  serve  as 
a  checklist  to  be  used  by  the  documenter  to  ensure  that  all  relevant 
subjects  have  been  discussed.    However,  it  is  not  as  extensive  or  inclu- 
sive as  FIPS  PUB  38  (8),  especially  in  terms  of  the  latter  guidelines' 
ability  to  capture  the  information  that  completely  describes  a  large 
complex  system. 

The  book  (9)  includes  the  material  of  (6)  and  (8)  with  some  variation 
and  extensions.    One  difference  is  the  definition  of  a  project  develop- 
ment cycle  (as  contrasted  to  a  software  life  cycle)  in  which  the  phases 
are  initiation,  analysis  (definition),  design  (detailed),  development 
(programming),^ implementation  (testing  and  conversion)  and  operation 
(operation  and  maintenance).    Although  here  limited  to  the  concern  of 
delivering  a  new  computer  software  system,  the  explicit  consideration 
of  implementation  is  important  when  structuring  documentation  for  the 
model  life  cycle. 

Of  extreme  importance  to  the  computer  model  user  and  developer  com- 
munities is  the  transferability  of  a  computer  program  to  another 
computer  hardware  system  and/or  the  ability  of  a  new  user  to  run  and 
interpret  the  results  of  the  program.    This  is,  of  course,  what  docu- 
mentation is  all  about,  and  the  DoD  manual  (6)  and  FIPS  PUB  38  (8)  are 
designed  to  facilitate  such  transfers  and  use.    We  have  noted  that  instal- 
lation should  be  concerned  with  stipulating  programming  procedures  to 
ensure  a  capability  of  running  and  maintaining  locally  developed  pro- 
grams.   A  broader  issue  is  how  to  document,  distribute  and  maintain 
computer  programs  that  are  to  be  transferred  and  used  by  many  instal- 
lations not  under  the  control  of  a  single  agency's  ADP  organization. 
We  discuss  next  two  approaches  to  this  issue. 
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The  URBAN  Mass  Transportation  Administration  (UMTA)  of  the  Department 
of  Transportation  sponsors  software  development,  including  computer 
models,  to  be  used  by  the  transportation  planning  community.  UNTTA  has 
developed  a  set  of  software  standards  that  must  be  adhered  to  by  all 
developers  of  programs  that  are  to  be  part  of  a  program  library  known 
as  the  UMTA  transportation  planning  system  (UTPS).    The  UTPS  is  used  by 
UMTA  and  others  in  planning  and  evaluating  multimodal  transportation 
systems.    These  standards  indicate  specific  documentation  and  mainten- 
ance requirements,  and  details  such  items  as  the  languages  of  the  compu- 
ter programs  and  standard  test  data  sets.    UMTA  defines  a  project's 
cycle  in  terms  of  four  development  steps:    general  functional  specifica- 
tions, detailed  technical  design  specifications,  computer  code,  and  test 
design  and  results.    UMTA  requires  documentation  material  for  each  step, 
with  a  fifth  step  being  a  review  by  UMTA  of  the  full  documentation  and 
approval  for  the  distribution  of  the  software  to  the  user  community. 
The  full  documentation  also  includes  a  user  guide,  data  set  descriptions, 
and  any  materials  related  to  briefings  or  training.    The  UMTA  software 
standard  describes  the  information  to  be  included  in  each  document,  but 
is  not  as  detailed  as  FTPS  PUB  38.    It  also  contains  document  preparation 
standards  such  as  format,  spelling,  etc.    The  program  writeup  standards 
are  designed  to  produce  documents  that  are  an  integral  part  of  the  pro- 
gram so  they  can  be  easily  modified  and  updated.    It  relies  heavily  on 
the  use  of  in-code  (embedded)  documentation  (via  comment  cards);  it  also 
employs  an  UMTA  designed  high-level  user  language  (USL)  for  program 
description.    The  USL  is  used  in  conjunction  with  the  structured  pro- 
gramming (top-down)  approach  for  software  development.    The  UMTA  soft- 
ware standards  and  their  implementation  include  elements  for  improving 
not  only  computer  programming  procedures,  but  the  specification  and  doc- 
umentation of  computer  models  as  well. 

The  second  approach  that  facilitates  the  transferability  and  use  of 
computer  programs  and  models  is  exemplified  by  the  DoD  Defense  Logistics 
Studies  Information  Exchange  (DLSIE)  and  its  catalog  of  logistics  models 
(23)  and  (24).    The  DLSIE  performs  the  following  functions: 

Acquires,  stores,  organizes  and  disseminates  information 
about  logistics  studies  and  models  (planned,  in-process 
and  completed)  and  other  logistics  documents,  journals 
and  books. 

Maintains  a  current  and  historical  inventory  of  all 
logistics  studies  which  may  be  of  significance  to  the 
research  and  management  of  logistics. 

Maintains  a  data  base  containing  relevant  information 
describing  logistics  modeling  efforts. 

Maintains  a  current  and  historical  inventory  of 
documentation  developed  pertaining  to  logistics 
modeling  efforts. 
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Develops,  publishes  and  distributes  appropriate 
documents  announcing  the.  information  accumulated 
from  reporting  activities. 

Provides  secondary  distribution  (hard  copy)  of  logistics 
research  and  management  information  to  authorized 
DLSIE  users. 

The  DLSIE  maintains  a  computerized  data  base  that  describes  each  logis- 
tics study  and  model.    This  data  base  is  used  to  prepare  the  catalogue 
of  models  (24)  by  generating  a  magnetic  tape  that  is  sent  directly  to 
the  Government  Printing  Office  to  be  read  by  a  linotron  (a  magnetic  tape 
to  film  negative  to  offset  printing  converter).    Individuals  can  also 
request  a  selective  dissemination  of  information  and  custom  biblio- 
graphies.   We  find  the  DLSIE  concept  and  its  implementation  an  impres- 
sive one  in  terms  of  a  researcher  or  user  being  able  to  find  out  what 
has  been  done  (and  by  whom)  in  a  particular  area  of  interest.  Granted 
it  is  a  rather  costly  operation,  but  in  many  areas  of  current  research 
and  development,  e.g.,  energy  and  economic  modeling,  a  similar  informa- 
tion system  would  pay  for  itself  in  better  usage  of  models,  and  infor- 
mation exchange  and  availability. 

Additional  approaches  relative  to  improving  the  transferability  of 
models  is  given  in  (57)  and  (58).    The  report  (57)  discusses  the  concepts 
of  modularization  as  applied  to  the  construction  of  large  and  complex 
models;  the  development  of  software  interfaces  that  enable  the  analyst 
to  link  previously  developed  analysis  and  data  base  programs  (e.g.,  sta- 
tistical and  report  generators),  and  the  linking  of  high  level  languages; 
and  the  development  of  wide- range  processors  that  support  the  set-up 
and  processing  of  models  that  use  different  methodological  approaches 
(e.g.,  systems  dynamics  and  econometric  models).    Modularization  is 
basically  an  application  of  the  top-down,  structured  approach  to  systems 
analysis  and  programming,  and  modular  concepts  are  becoming  standard  and 
accepted  by  many  agencies  (11)  and  (54).    The  use  of  software  interfaces 
and  wide-range  processors  is  not  as  advanced,  but  large-scale  modeling 
efforts  in  energy,  welfare  and  other  areas  will,  we  feel,  aid  in  the 
development  of  these  ideas. 

The  report  (58)  is  an  early  standard  developed  by  the  American  Nuclear 
Society  that  recommends  programming  practices  which  facilitate  the  inter- 
change of  computer  programs  prepared  for  scientific  and  engineering 
computationSo    Its  objective  is  to  simplify  conversion,  modification,  and 
use  of  computer  programs.    It  includes  such  recommendations  as  organizing 
the  program  into  reasonably  sized  subprograms,  and  the  minimization  of  the 
use  of  assembly  languages.    Report  (21)  is  a  companion  documentation 
standard. 
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In  this  section  we  have  reviewed  a  number  of  reports  that  were  developed 
as  guidelines  or  recommendations  to  improve  the  documentation  of  computer 
programs.    As  noted,  we  feel  that  some  of  these  ideas  and  documentation 
processes  are  applicable  to  the  documentation  needs  of  computer  models. 
Unlike  the  literature  on  the  documentation  of  computer  programs,  there 
are  no  official  guidelines  that  we  know  of  for  computer  model  documenta- 
tion.   Certainly,  the  programming,  user  and  other  documentation  guide- 
lines contained  in  FIPS  PUB  38  (8)  are  applicable  to  the  computer  aspects 
of  a  modeling  activity.    However,  it  is  our  contention  that  the  informa- 
tion requirements  of  computer  models,  especially  decision  models,  are 
not  satisfied  by  such  guidelines.    We  address  these  concerns  in  the 
next  sections  where  we  first  review  material  related  to  the  problem  of 
model  documentation,  and  then  discuss  an  approach  for  the  development  of 
computer  model  documentation. 


V.    Current  Guidelines  for  Computer  Model  Documentation 

As  evidenced  by  their  publications,  the  general  community  of  computer 
model  developers  and  model  researchers  has  not  been  overly  interested 
in  the  mundane  issues  of  model  documentation.    We  feel  that  this  disin- 
terest has  handicapped  severely  the  utility  and  acceptance  of  many  valid 
modeling  activities,  and,  in  turn,  allowed  some  invalid  models  to  be 
used.    The  situation  has  improved  over  the  last  few  years  due  to  studies 
conducted  by  and  the  pressures  due  to  the  GAO  and  other  Government  agen- 
cies (35),  (45),  (46),  (47)  and  (51),  writings  by  a  few  researchers  (37), 
(48)  and  (59),  and  the  difficult  documentation  requirements  of  important 
decision  models  (32),  (33),  (34),  (40),  and  (44).    Credit  should  be 
given  to  a  subgroup  of  computer  model  developers--simulation  modelers-- 
who  appear  to  have  had  a  continuing  interest  in  related  documentation 
procedures.    They  have  recognized  that  good  documentation  aids  in  estab- 
lishing the  validity  of  simulation  models  and  in  increasing  their  usage 
rates.    We  next  review  some  simulation  and  other  computer  model  documen- 
tation procedures. 

The  USAF  report  (17)  provides  guidelines  for  preparing  computer  program 
or  model  documentation.    It  pertains  to  the  specific  set  of  simulation 
models  and  computer  programs  that  are  developed  to  support  military 
analyses.    These  guidelines  describe  the  contents  to  be  included  in  a 
user  manual,  programmer  manual,  analyst  manual  and  a  summary  manual. 
The  report  suggests  that  user  and  programmer  manuals  are  the  minimum 
requirements  for  documenting  a  computer  model,  with  the  analyst  and 
summary  manuals  being  optional o    The  purposes  of  these  manuals  are  as 
follows. 

The  user  manual  must  include  the  information  necessary  for  the  user  to 
understand  and  substantiate  the  methodology  of  the  model  and  to  prepare 
accurately  the  model  inputs  and  interpret  the  model  outputs.    The  pro- 
grammer manual  must  provide  the  guiding  information  to  allow  a  program- 
mer to  modify  promptly  and  accurately  the  model  or  to  convert  it  for 
another  computer  system.    The  analyst  manual  must  be  designed  specifi- 
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callytomeet  the  needs  of  the  analyst  in  (1)  the  collection  and 
preparation  of  data  for  the  model,  (2)  interpretation  of  the  output 
results  which  may  be  expected  or  desired,  and  (3)  the  corrective  measures 
required  when  such  results  are  not  obtained.    The  summary  manual  contains 
a  brief  overview  of  the  model,  explaining  its  purposes  and  capabilities 
(the  why,  what,  and  how  of  the  model);  it  should  serve  as  a  handy  refer- 
ence for  a  quick  review  of  the  model. 

The  report  (17)  also  offers  guidelines  for  organizing  each  document  and 
for  document  maintenance.    It  suggests  that  new  manuals  should  be  pub- 
lished when  major  changes  are  made  to  the  model  or  when  a  series  of 
minor  changes  causes  change  to  more  than  one-fifth  of  the  documentation. 
The  guidelines  for  preparing  the  user,  programmer  and  analyst  manuals 
are  given  in  Appendix  B. 

The  guidelines  in  (17)  have  been  used  to  prepare  the  documentation  of 
certain  Air  Force  simulation  programs.    This  was  a  post-documentation 
effort  in  that  a  modeling  contractor  was  asked  to  review  already  existing 
and  running  programs  and  to  prepare  the  necessary  manuals.    The  detailed 
tables  of  contents  for  the  user,  programmer  and  analyst  manuals  for  this 
project  are  given  in  Appendix  C  (39). 

In  the  book  (41),  House  and  McLeod  comment  on  the  critical  issues  of 
documentation,  and  offer  a  most  comprehensive  discussion  on  how  to  devel- 
op proper  documentation  of  computer  models.    Although  their  examples  are 
based  on  the  needs  of  simulation  models,  it  is  felt  that  the  material  is 
applicable  to  computer  models  in  general. 

In  (41),  the  authors  differentiate  between  two  types  of  documentation 
information:  descriptive  and  technical.    For  descriptive  documentation, 
the  criterion  for  adequacy  is  an  affirmative  answer  to  the  question: 
"On  the  basis  of  this  documentation  alone,  would  it  be  possible  for 
anyone  reasonably  knowledgeable  in  the  field  to  determine  the  suitability 
and  availability  of  the  model  for  a  specific  use?"    While  for  technical 
documentation,  the  criterion  for  adequacy  is  an  affirmative  answer  to  the 
question:    "On  the  basis  of  this  documentation  alone,  would  it  be  pos- 
sible for  anyone  otherwise  competent  to  duplicate  and  run  the  model?" 
The  authors  note  that  descriptive  documentation  of  a  model  ls  usually 
available  (not  necessarily  the  kind  that  gives  a  positive  answer  to 
their  question,  however);    but,  technical  documentation  ls  in  short 
supply.    Such  technical  documentation  is  a  must  as  without  it  there  can 
be  no  evaluation  of  the  model  by  others,  no  model  transfer,  and,  in 
many  instances,  no  model  use  when  key  personnel  leave  the  project. 
They  emphasize  that  both  types  of  documentation  should  be  made  a  con- 
tractual requirement  by  the  funding  agency,  a  position  we  endorse. 

In  their  chapter  on  documentation.  House  and  McLeod  describe  the  pro- 
cedure used  to  document  the  complex  Strategic  Environmental  Assessment 
System  ^SEAS)  of  EPA  (66),  give  two  examples  of  documentation  formats 
used  to  produce  both  descriptive  and  technical  information  (due  to  McLeod 
(59)  and  Meadows  et  al .  [29),  and  present  a  documentation  checklist  to 
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be  used  as  a  means  of  ensuring  that  all  descriptive  and  technical  infor- 
mation is  included. 

In  the  article  (59j ,— ^McLeod  notes  that  documentation  of  the  development 
and  running  of  a  model  should  be  complete  enough  to  allow  for: 

a  peer  evaluation  of  the  work, 

the  reproducibility  of  the  runs  and  experiments, 

the  ability  of  others  to  build  on  the  work  reported 
instead  of  having  to  repeat  it,  or  to  start  it  over 
in  their  own  way. 

In  reviewing  tne  computer  model  documentation  issues  and  procedures  as 
described  in  the  Appendices,  the  reader  should  note  and  contrast  the 
differences  in  the  documentation  requirements  of  software  systems  and 
that  of  computer  models.    For  example,  the  documentation  procedure  of 
SEAS  includes  three  categories  of  documents:    system,  study,  and  pro- 
gramming.   The  system  documents  include  a  system  definition  document 
and  a  system  implementation  plan;  the  study  documents  allow  for  the 
research  nature  of  many  modeling  projects,  and  they,  along  with  the 
other  procedures  and  standards  of  SEAS,  are  designed  to  strike  a  balance 
between  the  flexibility  required  by  creative  model  building  and  the 
communication  of  information  essential  to  orderly  system  development 

(41)  and  (56).    Also,  for  computer  model  documentation,  there  is  a 
stronger  emphasis  on  user  (sponsor)  information  and  scope  of  user  invol- 
vement, model  assumptions  and  theoretical  basis,  validation,  and  the 
use  and  possible  experimental  nature  of  the  project. 

Central  to  the  use  of  a  mathematical /logical  model,  especially  those 
developed  as  aids  in  the2ipaking  of  policy  decision,  is  the  determination 
of  the  model's  validity.^    Validity,  in  turn,  is  Key  to  model  evaluation. 

Evaluation  is  a  process  by  which  interested  parties,  who  were  not  invol- 
ved in  a  model's  origins,  development  and  implementation,  can  assess  the 
model's  results  in  terms  of  its  structure  and  data  inputs  to  determine, 
with  some  level  of  confidence,  whether  or  not  the  results  can  be  used 
in  decision  making  (60).    The'sine  qua  non  of  such  third-party  evalua- 
tions is  the  availability  of  a  proper  set  of  documentation.    Many  authors 
address  this  concern;  see  discussions  in  (34),  (37),  (38),  (40),  (41), 

(42)  ,  (60),  (61),  (63),  and  (64)  for  evaluation  examples  and  related 
material . 


U   See  Appendix  D  for  article  (59), 

2/   Here  validation  tests  the  agreement  between  tne  behavior  of  the  model 
and  the  real  world  system  being  modeled;  and  verification  attempts 
to  ensure  that  a  model  (i.e.,  the  comouter  program)  behaves  as  the 
investigator  intended. 


23 


We  feel  that  the  evaluation  of  computer  models  will  be  a  continuing 
requirement,  and  any  documentation  process  for  computer  models  must 
also  address  the  needs  of  third-party,  independent  evaluation,    ihe  GAO 
evaluations  (34)  and  (40j  were  both  critical  of  the  status  of  the 
documentation  of  the  models  in  question.    The  criticisms  dealt  with  the 
documentation  not  being  up-to-date,  weak  in  program  maintenance  and  test 
analysis  documents,  no  test  plan,  and  general  lack  of  documentation. 
Documentation  deficiencies  are  rectified  sometimes  by  having  documenta- 
tion prepared  subsequent  to  the  evaluations  or  to  the  actual  use  of  the 
computer  model;  for  example  (30),  (3y)  and  (44).    Such  post-documentatio 
tends  to  be  well  structured  and  complete  and  can  aid  in  the  future  use 
and  maintenance  of  computer  models.    This  is  especially  true  for  less 
complex  and  more  directed  applications  like  police  patrol  car  allocation 
and  beat  design  models  ^30;  and  (68)  than  for  more  complex  models  like 
PIES  (44).    Reliance  on  post-documentation  to  supply  the  necessary  infor 
mation  for  model  use,  transfer,  evaluation  or  maintenance  is  a  poor  and 
a  much  more  costly  substitute  for  a  documentation  plan  initiated  at  the 
beginning  of  a  model's  development.    Concurrency  of  documentation  must 
be  maintained  and  sustained  along  with  the  other  major  modeling  activi- 
ties. 

txtending  computer  model  documentation  to  include  the  needs  of  evalua- 
tion requires  the  development  and  supplying  of  information  that  is 
beyond  that  usually  documented  or  even  requested.    If  such  information 
is  not  generated  during  the  phases  of  a  modeling  project,  then  it  is  our 
contention  that  the  modeling  process  employed  was  incorrect  and  the 
project  management  was  lacking.    A  basic  hypothesis  that  should  apply  to 
the  documentation  process  of  a  computer  model  is  that  upon  completion  of 
the  model  and  after  its  Implementation  and  maintenance  phases  are 
entered,  documentation  should  have  been  produced  that  furnishes  all 
information  that  would  enable  an  independent  review  and  assessment  to 
be  performed.    This  set  of  Information  includes  the  software  documenta- 
tion described  in  07;,  (20),  (34),  (35j,  (37),  (3y),  ^40;,  (4l),  {^Z) , 
(51 j,  (b9),  (6u),  and  (61;.    In  the  next  section,  we  shall  address  the 
overall  process  for  computer  model  documentation  that  is  based,  in  part 
on  this  hypothesis.    But  before  doing  that,  we  conclude  this  section's 
review  of  computer  model  documentation  Issues. 

In  addition  to  those  noted  above,  other  investigators  have  described 
what  they  feel  are  the  necessary  information  elements  that  must  be 
contained  in  computer  model  documentation.    Brewer  (37)  offers  the 
following  as  a  minimal  set  of  information  required  to  operate  a  computer 
model : 

t        Program  listing. 

t        Variable  listings,  definitions. 

•  Flow  charts. 

•  Verbal  description  of  the  program. 

•  Operator ' s  manual . 

•  Programmer's  manual. 

•  bummary  of  theoretical  bases  of  the  model. 
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Data  reduction  methods  and  techniques  used  to  specify 
the  model's  relationships  and  to  assign  input  values 
to  parameters. 

Cost  data  related  to  production,  operation  and  updating. 
Listing  of  personnel  invdlived  in  all  phases  of  the  model's 
existence. 

Existence  and  history  of  external  professional  review, 
including  findings  and  remedial  steps  taken. 
Utility  analyses  types  of  use,  frequency  of  use,  cost, 
assessment  of  outcomes. 


A  similar  listing  is  offered  by  Shubik  and  Brewer  (45);  there  the  recor- 
ding of  the  information  associated  with  those  persons  involved  with  the 
model  is  stressed.    For  example,  atmodel  initiation— who  wanted  it  built 
and  for  what  reasons;  during  model  production--who  was  the  master  modeler, 
who  was  on  the  model  team  and  what  validation  procedures  did  they  use; 
during  model  operations — what  type,  if  any,  external  review  was  made 
of  the  model  and  what  was  its  results;  and  during  use--who  used  it, 
when,  and  with  what  purpose  and  outcome.    We  feel  that  the  capturing 
of  this  information  is  extremely  important.    A  model  tends  to  be  an 
ever-changing  entity.    Its  form  and  outcomes  are  reflections  of  its 
developers  and  users.    Only  by  having  full  knowledge  of  their  involvement 
can  we  understand  a  particular  modeling  process. 

In  an  attempt  to  standardize  a  modeling  reporting  format.  House  (42) 
structured  a  questionnaire  approach  that  is  presented  below.    It  is 
suggested  that  each  model  builder  be  required  to  deliver  a  completed 
qliestiohnaire  as  a  final  product.    Given  such  a  set  of  questionnaires, 
then  a  model  information  exchange  system  could  be  initiated  similar  to 
the  DoD  logistics  model  information  system  {26). 

Model  Reporting  Format  (42) 

A.  Basic  description  of  model 

1.  Name  or  title  of  the  model, 

2.  Developer(s j . 

3.  Agency  or  company. 

4.  Sponsor;  purpose  or  objective  of  sponsor. 

5.  When  developed? 

6.  Where  developed? 

7.  How  much  time  and  money  did  it  take  to  develop? 
Is  model  proprietary? 

8.  Developed  separately  or  as  part  of  a  larger  study? 

9.  Where  used  and  in  use;  frequency  of  use? 

B.  Subject  matter  of  model 


1. 
2. 
3. 


Major  purpose  (objective)  of  the  model. 
Scope  (subject  matter)  of  the  model. 
Was  the  model  based  on  a  particular  description 
or  theory?    If  so,  what  one? 
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4. 


5. 


Does  the  model  usage  require  knowledge  of  a 
specific  discipline? 

How  is  the  model  different  from  other  similar  models? 


C. 


Modeling  technique 


1.  m  an  analytical  sense,  what  type  of  model  is  it? 

2.  Does  the  model  use  any  standard  packages  (e.g. 
linear  programming,  statistical,  etc.)? 

3.  Was  the  model  developed  from  another  model? 
If  yes,  what  one(s;  and  how? 

4.  Is  its  structure  clear?  its  variables? 

5.  What  are  the  data  requirements  of  the  model? 

6.  Does  the  model  receive  any  data  from  other  models? 
Is  it  required  or  optional? 

7.  What  constraints  does  the  model  have  (e.g.  modes 
of  transportation,  city  size,  spatial,  etc.)? 

D.  Computer  aspects  of  the  model 

1.  What  computer  language(s;  is  the  model  written  1n? 

2.  What  machine (sj  is  It  programmed  for? 

3.  How  much  time  does  it  take  to  run? 

4.  How  large  is  the  model  (lines  of  code,  core  to  run,  etc.)? 

5.  HOW  many  parameters  does  the  model  require? 

E.  Validation  of  the  model 

1.  Has  the  model  been  validated?  How? 

2.  is  the  model  documented?  How  well? 

3.  Has  the  model  been  critiqued  or  appraised?  By  whom? 
At  what  point? 

4.  Has  there  been  a  sensitivity  analysis  of  any  type 
performed  on  the  model?  If  so,  by  whom? 

5.  Can  it  be  used  from  the  current  documentation? 


1.  If  asked,  how  would  you  demonstrate  the  utility  of 
the  model?    Have  you? 

2.  With  whom  should  one  get  in  touch  to  discuss  use  of 
the  model? 

3.  How  much  would  it  cost  to  transfer  the  model? 

4.  Are  the  model  relationships/parameters  easy  for  the 
user  to  change? 

5.  Have  there  been  any  papers  given  or  written  on  the  model? 
Where'?    When?    By  Whom? 

6.  Is  the  output  of  the  model  special  purpose  or  is  it 
designed  for  a  general  audience? 


Has  it  been? 


F. 


Model  use  and  transferability 
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The  report  (43)  describes  the  activities  of  a  workshop  on  energy 
policy  models.    It  noted  that  a  call  for  better  documentation  was 
repeated  by  nearly  every  speaker.!/    In  fact,  the  existence,  timeliness, 
completeness,  readability,  dissemination  and  purposes  of  most  docu- 
mentation were  challenged  or  criticized  by  the  workshop  participants. 
The  report  further  notes,  however,  that  the  sanctity  of  belief  in  good 
documentation  was  challenged  by  counter  charges  that  current  documen- 
tation is  not  read;  and  despite  the  rhetoric,  there  is  no  financial 
support  for  documentation  preparation  because  users  are  not  interested 
in  having  or  reading  documentation.    These  counterclaims  do  have  some 
validity. 

It  is  incumbent  upon  all  model  sponsors,  developers  and  users  to 
confront  the  documentation  requirements  at  the  inception  of  a  model. 
If  one  sponsor  wishes  to  specify  and  support  a  certain  level  of  docu- 
mentation, then  this  should  be  stated,  agreed  to  and  written  into  the 
project  files.    For  example,  the  costs  of  documentation  are  increased 
if  an  objective  of  the  model  is  to  have  it  be  transferable  to  other 
users.    If  this  is  not  the  case,  then  the  sponsor  and  user  should  so 
stipulate  and  indicate  the  documentation  required  for  the  user's  local 
environment.    In  this  way,  the  modeling  community  is  forewarned  and 
criticisms  of  the  model  for  what  it  was  not  designed  to  accomplish 
would  be  unwarranted. 

In  a  similar  vein,  the  model  developer  must  specify  a  documentation 
plan  that  fits  the  resources  allocated  and  the  objectives  of  the  model. 
If  the  documentation  objectives  cannot  be  accomplished,  then  the 
developer  must  face  and  resolve  this  problem  with  the  sponsor  and  user 
early  in  the  project  life  cycle. 

In  the  report  (49),  Chaiken  notes  that  the  documentation  of  a  model 
plays  many  roles.    The  existence  of  a  user  manual  is  an  absolute 
prerequisite  for  dissemination  of  a  model  to  recipients  who  do  not  have 
technical  assistance  from  the  model  developers.    But,  the  availability 
of  this  manual  does  not  guarantee  dissemination.    Some  documentation 
can  discourage  dissemination  in  that  It  honestly  states  the  difficulties 
and  cost  of  transfering  and  using  the  model.    The  developer  in  so 
stating  does  a  service  to  the  community  of  possible  users. 

Documentation  also  alerts  a  potential  user  to  the  availability  of  the 
model,  and  for  this  purpose,  a  clear  and  brief  executive  summary  is 
of  more  value  than  the  user  manual.    Chaiken  also  notes  that  the 
availability  of  an  annotated  program  listing  is  evidently  reassuring  to 
a  potential  user,  even  though  few  users  ever  read  the  listings  with  care. 
Having  the  listing  suggests  that  the  program  Is  finished  and  not  subject 
to  repeated  modifications,  even  though  this  may  not  be  true.    It  also 

17   Also  see  Fromm  et  al.  (46)  for  results  of  a  survey  that  discusses 
model  documentation  quality,  availability  and  how  documentation 
relates  to  model  use. 
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indicates  that  the  program  Is  not  proprietary,  and  that  the  developer 
has  confidence  in  the  model  as  it  is  available  to  be  read  by  others. 

As  a  final  item  in  this  section,  we  coiraient  on  recent  investigations 
directed  towards  the  development  of  software  and  model  documentation 
aids  and  languages;  see  (4),  (5j,  (l2),  (15),  (54j,  (57),  and  (65).  A 
few  researchers  in  the  areas  of  simulation,  models,  large-scale  systems 
models,  complex  software  developments  and  their  documentation  have 
proposed  and  constructed  special  procedures  for  designing  and  communica- 
ting their  products.    For  example,  the  DELTA  (Development  of  Language 
Tools  for  Administration  and  Research)  project  (4)  has  as  its  objectives 
the  development  of  a  set  of  concepts  and  a  language  for  descriptions  of 
systems  and  communication  about  systems.    The  report  (4;  describes  a 
way  of  analyzing  systems,  defined  as  DELTA-structured  systems,  and  a 
DELTA-language  for  communication  about  such  systems,    ihe  authors  feel 
that  current  computer  programming  languages,  simulation  languages  and 
natural  languages  are  not  appropriate  for  describing  complex  systems. 

In  the  report  (5),  Nance  proposes  the  development  of  a  simulation  model 
specification  and  documentation  language  (SMSDL).    As  proposed,  the 
SMSDL  would  facilitate  model  specification  and  documentation,  describe 
the  model  at  the  high  and  low  (top-down)  levels,  be  applicable  to 
diverse  problem  areas,  be  compatible  with  present  simulation  languages, 
and  aid  in  validating  and  verifying  a  simulation  model. 

We  feel  that  systems  and  modeling  languages  are  but  a  part  of  the  model- 
ing process,  and  they  should  not  be  looked  at  as  being  the  procedural 
way  to  resolve  the  current  difficulties  in  producing  documentation  that 
will  satisfy  the  diverse  needs  of  the  modeling  community.    It  is  our 
opinion  that  these  languages  will  be  more  aids  to  the  analyst  and 
programmer  in  doing  their  work  and  not  be  communication  devices  that 
improve  the  user/decision-maker  understanding  of  a  model  and  the  modeling 
process.    As  modelers,  we  need  to  describe  the  user/decision-maker  prob- 
lem environment  and  this  may  be  done  best  by  a  special  language.  But, 
we  cannot  impose  such  a  language  upon  the  users.    They  are  not  modelers 
and  their  interests  lie  elsewhere. 


An  approach  related  to  the  improvement  of  documentation  is  the  use  of 
computer-aided  documentation  programs  (12),  (15),  and  (65).    The  report 
(15)  describes  a  software  design  and  documentation  language  (SDDL)  that 
is  a  communications  medium  that  supports  the  design  and  documentation  of 
complex  software  applications.    The  SDDL  provides  a  design  and  documen- 
tation language,  a  processor  for  converting  design  specifications  into  a 
machine  reproducible  document,  and  a  methodology  for  using  the  language 
and  processor.    Such  procedures  enable  the  system  designers  and  program- 
mers to  maintain  the  currency  of  their  efforts,  and  to  relieve  them  of 
some  of  the  tedious  aspects  of  document  preparation,  e.g.  flow-charting, 
formatting,  up-dating.    These  efforts,  plus  research  and  development  in 
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special  modeling  languages,  should  be  encouraged  in  that  they  can  contri- 
bute new  tools  for  understanding  and  performing  the  modeling  process. 
However,  we  must  realize  that  the  ultimate  needs  of  computer  model  docu- 
mentation are  not  in  new  languages,  or  even  a  new  universal  language,  and 
are  not  in  special  aids  and  devices. 

The  furthering  of  the  use  of  models  and  the  benefits  to  be  gained  by 
their  proper  use  can  be  done  only  by  removing  the  technical  barriers  that 
now  separate  the  model  developers  from  the  model  users.    We  cannot  expect 
the  latter  group  to  be  conversant  with  system  languages,  computer  method- 
ology,   etc.    We  must  communicate  the  bases  and  results  of  our  modeling 
efforts  to  this  group--the  user,  decision  maker,  sponsor  group--via 
documents  that  are  understandable,  inclusive,  and  timely.    And  this  must 
be  accomplished  using  a  natural  language— here  English--and  a  documenta- 
tion process  supported  by  both  model  sponsors  and  developers. 

VI .    An  Approach  for  the  Development  of  Computer  Model  Documentation 

In  this  section  we  describe  an  approach--a  disciplined  approach--to 
resolving  the  problem  of  obtaining  proper  documentation  of  a  computer 
model.    In  this  context,  proper  documentation  provides  specific  and 
detailed  information  that  is  organized  and  presented  in  a  manner  that 
will  satisfy  the  needs  of  each  segment  of  a  model's  audience.  This 
audience  consists  of  the  model's  sponsors  and  users  (possibly  non-tech- 
nical ly  oriented);  the  model's  analysts,  programmers,  and  computer 
operators;  other  users;  other  analysts,  programmers  and  computer  oper- 
ators; and  independent  model  evaluators.    Although  we  are  concerned 
mainly  with  the  documentation  requirements  for  large-scale  decision 
models,  our  approach  is  applicable  to  all  computer  models. 

We  base  our  approach  on  the  following  assumptions: 

t        Computer  program  and  software  documentation  of  a 

model  must  follow  the  guidelines  of  FIPS  PUB  38  (8) 
and  its  future  amendments. 

•  Computer  model  documentation  must  provide  sufficient 
information  that  would  enable  an  independent  review 
and  evaluation  of  the  model  to  be  performed. 

t        Computer  model  documentation  must  describe  all 

historical,  technical,  developmental,  maintenance 
and  implementation  aspects  of  the  model,  including 
assumptions,  implications  and  impact  of  using  the  model 
in  a  decision  situation. 

•  The  organization  of  a  modeling  project  must  include  a 
formal  documentation  activity  with  stated  objectives 
and  assignment  of  resources  (personnel,  funds,  time) 
for  their  accomplishment. 
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•      As  a  means  of  managing  the  documentation  activity  of 
a  modeling  project,  documentation  must  be  produced 
that  corresponds  to  the  phases  of  the  model  life  cycle, 
and  the  production  and/or  maintenance  of  the  documenta- 
tion must  be  concurrent  with  the  time  span  of  each  phase. 

The  rationale  behind  these  assumptions  is  founded  on  the  following  con- 
siderations.   The  FIPS  PUB  38  are  based  on  guidelines  and  practices  in 
Federal  agencies  and  other  institutions  and  were  developed  after  years 
of  study  by  many  Government  computer  experts.    The  FIPS  guidelines  are 
being  adopted  by  many  agencies  and  we  envision  their  becoming  the 
"standard"  guidelines.    Also,  the  National  Bureau  of  Standards  is 
continually  monitoring  and  updating  these  guidelines.    Thus,  we  see 
no  need  to  develop  other  computer  program  or  software  documentation 
procedures.    The  ten  FIPS  PUB  38  documents  (that  were  described  briefly 
in  Section  IV)  are  adequate  for  capturing  all  computer  aspects  of  a  model- 
ing project.    When  developing  these  and  other  documents  for  a  modeling 
project,  there  will  be  redundancies.    We  need  to  bear  in  mind  that  some 
information  must  be  presented  to  different  audiences  and  most  of  the 
documents  must  stand  alone.    In  the  final  analysis,  the  full  set  of 
information  must  be  presented  in  the  right  place  and  at  the  right  time 
to  the  corresponding  interest  group.    To  err  on  the  side  of  redundancy 
to  assure  completeness  is  the  correct  thing  to  do. 

If  a  modeling  project  is  managed  correctly  and  conducted  using  proper 
modeling  methodologies,  then  the  project  staff  would  have  had  to  invesr 
tigateall  issues  relating  to  the  model's  assumptions,  analytical  basis, 
development  and  use.    If  the  results  of  these  investigations  are  coupled 
with  the  results  of  the  computer  implementation,  and  all  such  information 
is  recorded,  then  an  independent,  third  party  review  team  would  only 
need  to  read  this  information  to  determine  the  adequacy  of  the  model. 
The  team  would  not  have  to  make  sensitivity  analyses,  determine  if  the 
data  are  correct,  etc.,  as  the  project  management  would  have  demonstrated 
that  these  matters  have  been  taken  care  of.    The  review  team  would  then 
need  to  be  concerned  only  with  the  reproducibility  of  the  results  and 
with  spot-checking  to  guarantee  the  veracity  of  the  documentation.  We 
believe  that  every  modeling  project  should  be  conducted  as  if  an  external 
review  of  the  model  and  its  results  will  be  made,  if  not  in  fact,  then  in 
concept.    If  this  were  the  case,  then  what  was  just  stated  above  would  be 
the  norm  for  model  review  teams.    However,  we  recognize  that  this  will 
not  be  true  for  most  modeling  projects,  if  for  no  other  reasons  than  the 
lack  of  discipline  within  most  modeling  management  and  the  additional 
costs  required.    Thus,  we  only  assume  that  the  model  documentation  will 
be  complete  enough  for  an  external  team  to  be  able  to  perform  additional 
tests  and  analyses  to  evaluate  the  computer  model. 

The  final  form  of  a  computer  model  is  a  function  of  its  origins  (who 
wanted  it  and  why)  and  its  total  evolutionary  history.    The  true  under- 
standing of  a  model  and  its  possible  utility  can  only  be  accomplished  by 
the  documentation  of  this  history.    Hence,  we  think  that  it  must  be 
recorded. 
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As  previously  described,  the  overall  value  of  much  of  the  Government's 
modeling  activity  is  debatable.    A  good  portion  of  this  disutility  is 
due  to  the  lack  of  proper  documentation.    Any  proposals  made  here  and 
elsewhere  for  producing  documentation  of  computer  models  are  worthless 
unless  the  modeling  community  recognizes  that  documentation  is  the  key 
to  model  utility.    Project  managers  must  insist  upon  having  documentation 
objectives  and  resource  support  from  their  sponsors.    The  delimiting  of 
documentation  is  the  proper  tning  to  do,  1f  for  no  other  reason  than  to 
protect  the  project  management  from  criticism  by  those  who  want  to 
use  the  model  beyond  the  project's  requirements. 

Finally,  documentation  must  be  an  integral  part  of  all  phases  of  a  model- 
ing project  and  that  documentation,  to  be  of  real  value,  must  be  an 
output  of  the  ongoing  project.    Post  documentation  of  models  must  be 
el iminated--it  is  inefficient,  it  usually  relies  on  personnel  who  were 
not  involved  in  the  model's  origins  and  development,  and  it  is  not 
timely.    Documentation  must  be  related  to  the  modeling  life  cycle  and 
all  personnel  who  intersect  this  life  cycle  must  recognize  their  duty 
towards  the  model's  documentation. 

Any  approach  for  documenting  computer  models  must  not  be  rigid  and 
must  allow  for  the  specific  needs  of  the  project.    Thus,  flexibility  and 
innovation  in  documentation  Is  encouraged.    But,  this  should  not  be 
interpreted  as  a  license  to  just  get  by  and  do  the  minimum.    Also,  model 
documentation  should  describe  not  only  what  the  model  can  do,  but  also 
what  the  model,  as  designed,  cannot  do.    If  a  model  is  experimental  and 
should  not  be  used  in  an  operational  or  decision  setting,  this  should  be 
stated  explicitly.    The  above  comments  are  just  another  way  of  stating 
that  professionalism  must^  be  a  part  of  the  documentation  activities  of 
all  computer  modeling  projects. 

As  noted  in  the  previous  sections,  computer  programmers,  model  builders 
and  systems  analysts  tend  to  think  of  their  efforts  in  terms  of  major 
phases,  stages  or  steps  (8),  (ly),  l51;,  (bO),  and  (67).    in  many  pro- 
jects, the  activities  and  resources  are  divided  formally  into  such  seg- 
ments and  progress  and  expenditures  are  accounted  for  by  segment.  It 
is  recognized,  of  course,  that  a  modeling  project's  phases  overlap  In 
both  time  and  resources,  and  a  project  develops  along  parallel  phases, 
not  serially.    In  any  event,  we  believe  that  good  model  management 
practice  requires  the  production  of  documentation  to  be  related  to  a 
model's  life  cycle  phases.    This  concept  is  just  an  extension  of  the 
FIPS  PUB  38  software  cycle  -  document  production  process  to  the  modeling 
environment.    For  the  documentation  needs  of  a  model,  the  software  life 
cycle  is  too  limited  and  aggregated  and  must  be  extended  and  refined, 
especially  for  complex  decision  models. 

We  next  define  an  appropriate  phase  segmentation  of  a  computer  model 
life  cycle,  followed  by  a  discussion  of  the  information  to  be  produced 
in  each  phase  and  the  associated  documentation.    It  is  not  our  purpose 
here  to  produce  a  FIPS  PUB  38-1  ike  set  of  guidelines  for  computer 
models.    However,  what  is  offered  below  can,  we  feel,  be  used  as  a 
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basis  for  any  such  development.    We  emphasize  that  these  model  life 
cycle  phases  are  interdependent,  do  not  necessarily  coincide  with  fixed 
time  periods,  and  are  just  convenient  groupings  of  project  activities 
that  can  be  related  to  the  documentation  requirements  of  a  modeling 
project. 

COMPUTER  MODEL  LIFE  CYCLE  PHASES 

t        Embryonic:    during  this  pnase,  the  to  be  sponsor/user 
contemplates  the  application  of  modeling  methodology 
to  aid  in  resolving  a  problem  area,  i.e.,  an  idea  nas 
been  hatched. 

•  Feasibility:  during  this  phase,  the  problem  is  defined 
and  delimited,  and  specific  approaches  for  solving  the 
problem  are  conceived  and  evaluated,  i.e.,  an  investiga- 
tion and  decision  as  to  whether  the  idea  can  and  should 
be  developed  further  is  undertaken. 

•  Formulation:    during  this  phase,  the  analytical  basis  of 
the  selected  solution  approach  is  developed,  i.e.,  the 
idea  is  represented  in  terms  of  a  model. 

t        Data:    during  this  phase,  the  Information  requirements 
to  support  the  model  and  its  development  are  determined, 
and  activities  for  the  collection  and  analysis  of  the  data 
are  Initiated,  i.e.,  data  that  describe  and  support  the 
model  are  determined  to  be  available  and  are  collected. 

•  Design:    during  this  phase,  the  analytical,  data  and 
computer  requirements  are  integrated  into  a  set  of 
system  specifications  for  resolving  the  problem,  i.e., 
the  user's  problem  requirements,  as  described  by  the 
model,  are  combined  with  computer  and  programming 
approaches  to  produce  a  viable  technical  solution. 

«        Software  Development:    during  this  phase,  the  design 
specifications  are  converted  into  tested  and  operating 
software,  i.e.,  the  design  is  processed  through  the 
four  stages  of  the  FIPS  PuB  38  software  development 
phase — definition,  design,  programming,  test— to 
produce  a  verified  computer  system. 

•  Va I idation:    during  this  phase,  a  validation  or  acceptance 
test  plan  Is  developed  and  carried  out  to  validate  data 
extensions  ie.g.,  parameters,  forecasts),  the  model  and 
its  subcomponents,  and  the  verified  computer  system. 

The  plan  should  include  agreed  upon  test  cases  or 
scenarios,  sensitivity  analyses,  tests  for  robustness, 
historical  validity,  etc.,  i.e.,  the   model,  as  repres- 
ented by  tne  computer  system,  is  tested  against  speci- 
fied user  requirements  and/or  system  objectives  to 
determine  user  acceptability. 
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•  Training  and  Education:    during  this  phase,  the  user 
groups  involved  in  the  future  use  of  the  model--decision 
makers,  analysts,  computer  programmers,  computer 
operators,  data  collectors,  etc. --are  trained  in  appro- 
priate aspects  of  the  computer  system,  including  main- 
tenance of  the  model  and  system.  I.e.,  a  complete 
training  program  must  be  developed  and  given  (this 

is  most  important  if  the  computer  system  was  developed 
by  an  external  organization). 

•  Instal lation:    during  this  phase,  the  verified  and  vali- 
dated computer  model  is  installed,  tested  and  operated 
on  the  user's  computer,  i.e.,  if  the  computer  system 
used  for  development  and  test  is  not  the  user's  system, 
then  an  instal lation  plan  must  be  developed  and  carried 
out  to  ensure  compatabllity  of  the  computer  systems  and 
reproducibility  of  results. 

•  Implementation:    during  this  phase,  the  user  organiza- 
tion integrates  the  computer  system  into  its  operating 
environment  and  procedures  are  developed  for  generating 
and  requesting  specific  computer  analyses,  and  inter- 
preting and  using  tne  results,  i.e.,  the  idea  has 
matured  into  a  verified  and  validated  computer  model  and 
the  model  is  made  part  of  the  organization's  (decision) 
activities. 

•  Maintenance  and  Update:    during  this  phase,  a  process 
for  maintaining  the  computer  model  is  developed  and 
implemented,  including  modifications  to  the  model, 
programming  changes,  input/output  procedures,  data 
and  parameter  changes,  file  maintenance,  etc.,  i.e., 
activities  are  structured  and  implemented,  and  person- 
nel and  funds  allocated  to  ensure  that  the  model  will 
continue  to  represent  the  user's  view  of  the  problem 
and  its  environment. 

•  Evaluation  and  Review:    during  this  phase,  a  procedure 
is  established  that  provides  for  independent  third- 
party  assessments  of  the  model  and/or  periodic  reviews 
by  the  user,  i.e.,  depending  on  the  importance  of  the 
model  in  a  decision  environment,  a  plan  for  a  detailed 
independent  assessment  is  developed  and  implemented,  or 
the  user  establishes  internal  review  team  procedures 

to  ensure  that  the  model  is  updated  properly  and  is  still 
required  by  the  organization. 

•  Documentation  and  Dissemination:    during  this  phase,  a 
documentation  plan  is  developed  and  implemented  for 
recording  of  the  results  of  all  other  phases,  i.e., 
documentation  objectives  are  agreed  to,  the  requirements 
of  specific  documents  are  ;stated,  arid  the  documehts  are 
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produced.    The  documentation  phase  begins  during  the 
embryonic  phase  and  continuous  throughout  the  model's 
life  cycle.    If  appropriate,  a  plan  for  disseminating 
documents  and  information  on  the  structure,  utility  and 
use  of  the  mode!  is  also  initiated  and  implemented. 

Before  describing  the  documentation  requirements  for  the  above  model  life 
cycle  phases,  we  next  recapitulate  our  view  of  what  successful  model  doc- 
umentation  must  include.    A  major  reason  why  models  are  not  utilized 
properly  or  utilized  at  all  is  due  to  incomplete  and  out-of-date  documen- 
tation.   Our  approach  to  correcting  this  flaw  is  to  be  information  greedy, 
i.e.,  within  reason,  require  all  participants  to  keep  informal  and  cur- 
rent records  of  their  project  activities.    This  information  is  made 
available  to  the  documentation  staff  so  they  may  use  it  for  or  combine 
it  with  the  project's  formal  documentation.    We  are  describing  here  a 
general  documentation  approach  that  is  directed  towards  improving  the  value 
of  complex  decision  models.    The  approach  can  apply  also  to  "simple" 
models.    The  principle  of  information  overkill  should  not  be  abandoned 
because  the  model  is  simple  unless  a  purposeful  decision  is  made  to 
reduce  information. 

In  a  modeling  milieu,  we  must  be  overly  concerned  with  being  able  to  know 
and  understand  the  problem  situation  and  its  origins,  the  assumptions  of 
the  modeling  approach,  the  decision  environment  and  the  user  objectives, 
the  validity  of  the  model  and  data,  where  the  model  can  be  used,  etc.  We 
have  found  that  much  of  this  information  is  never  recorded,  and  sometimes 
it  is  never  known  or  stated  explicitly.    If  a  model  has  not  been  validated 
or  cannot  be  validated,  it  should  be  so  stated.    If  a  programmer  ran  a 
verification  check  of  a  subroutine  during  the  midnight  shift,  it  should 
be  so  stated.    If  the  user  has  ruled  out  a  solution  alternative  or  imposed 
other  restrictions  that  influence  the  form  of  the  model,  it  should  be  so 
stated.    The  acceptability,  evaluation  and  future  utility  of  a  model  can 
be  determined  only  by  a  complete  documentation  record  that  relates,  not 
just  what  has  been  done,  but  what  has  been  omitted  and  why.    Thus,  the 
plans  and  activities  required  by  the  above  model  life  cycle  phases  need 
to  be  developed  explicity  and  their  results  recorded.    Much  of  the  infor- 
mation can  be  contained  in  informal  analyst  and  programmer  notebooks, 
memoranda  and  working  papers.    The  documentation  phase  should  include 
activities  for  gathering  and  cataloguing  such  informal  information. 

We  describe  next  the  formal  documents  that  should  be  developed  for  any 
computer  model  project.    Depending  on  the  scope  and  ultimate  use  of  the 
model,  some  of  these  documents  can  be  eliminated  or  combined.    In  any 
event,  the  user/sponsor  and  the  model  developer  must  conclude  an  agree- 
ment as  to  the  documents  produced,  their  contents,  uses  and  audiences. 
In  what  follows,  we  shall  sketch  out  the  information  to  be  recorded  in 
each  document,  recognizing  that  we  are  not  aiming  for  completeness  in  this 
study.    Our  purpose  has  been  to  review  model  documentation  proposals  and 
to  give  some  direction  to  future  documentation  guideline  efforts. 

Some  of  the  documents  are  direct  products  of  a  particular  phase,  while 
others  contain  information  from  a  number  of  phases  or  are  the  outcome  of 
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the  total  project.    The  form  of  the  documents  can  range  from  a  few  pages 
to  detailed  manuals.    We  note  again-  that  it  is  incumbent  upon  all  persons 
involved  in  the  model's  development  to  maintain  current  records  of  their 
activities  as  they  pertain  to  the  model's  specifications,  assumptions, 
analytical  basis,  computational  requirements  and  testing,  validation, 
data  sources  and  collection,  implementation  and  maintenance. 

COMPUTER  MODEL  DOCUMENTS 

NEEDS, DESCRIPTION:    Embryonic  Phase 

A  discussion  of  the  origins  of  the  model  idea  including  who  initi- 
ated it,  why,  who  are  to  be  the  users  and  what  are  their  needs,  extent  of 
problem,  general  description  of  problem  and  decision  environment,  why 
modeling  was  considered,  preliminary  feasibility  considerations,  other 
solution  approaches,  why  a  computer  model,  impact  of  problem  and  solution, 
what  is  expected  from  solution,  how  model  and  solution  are  to  be  used, 
etc.    This  is  a  historical  document  that  can  be  formed  from  memoranda, 
notes,  working  papers,  records  of  meetings,  and  possibly  user  or  developer 
proposals. 

FEASIBILITY  STUDY:    Feasibility  Phase 

A  report  that  describes  the  background,  purpose,  scope,  organiza- 
tions, and  participants  involved  in  the  study;  definition  of  the  problem 
and  issues  and  objectives,  requirements  to  be  met;  organizations,  func- 
tions and  systems  examined;  solution  alternatives  with  costs  and  benefits; 
recommended  computer  model  solution  and  justification,  plan  of  action  and 
schedule  of  activities;  resource  requirements  (personnel,  funds,  computer, 
facilities);  etc.    This  is  a  historical  document  that  describes  the  pro- 
cess used  to  determine  that  a  computer  model  can  and  should  be  developed 
to  resolve  the  problem.    It  should  describe  the  role  of  the  model  in  the 
user  organization,  who  the  users  are,  and  the  range  of  decision  situations 
to  be  evaluated  by  the  model. 

MODEL  FORMULATION  DESCRIPTION:    Formulation  Phase 

This  report  describes  the  complete  details  of  the  mathematical/logical 
model;  the  theoretical  and  analytical  rationale  for  its  form  in  terms 
of  the  problem  definition;  assumptions,  hypotheses  and  restrictions; 
parameter  estimation  procedures;  general  data  requirements;  computational 
and  numerical  analysis  requirements;  computer  resources  required;  ap- 
proaches and  tests  for  validating  the  model;  sensitivity,  robustness  and 
other  evaluations  required;  restrictions  on  the  use  and  range  of  the  model, 
etc.    This  is  an  operational  document  maintained  throughout  the  model  life 
cycle.    The  model  structure  is  usually  modified  over  time  and  a  procedure 
for  updating  the  description  of  the  model  must  be  initiated. 

DATA  REQUIREMENTS  DESCRIPTION:    Data  Phase 

This  report  describes  the  detailed  data  needs  as  required  by  the 
model ;  data  sources;  the  process  for  obtaining  the  data;  experiments, 
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data  collections  and  surveys  to  be  performed;  organizational  and  individ- 
ual responsibilities  for  obtaining,  updating,  and  processing  the  data; 
numerical  and  forecasting  techniques  to  be  used  for  parameter  estimation; 
data  validation  procedures;  acceptable  data  ranges;  data  input  procedures 
to  the  computer  model,  etc.    This  is  an  operational  document  that  is  main- 
tained throughout  the  model  life  cycle. 

DESIGN  SPECIFICATION:    Design  Phase 

This  report  is  the  major  document  that  summarizes  the  results  of 
the  preliminary  analyses  and  details  how  the  model,  data  and  computer 
aspects  of  the  problem  solution  are  to  be  integrated  into  an  operational 
computer  system.    It  is  here  that  the  computer  requirements,  programming, 
and  software  specifications  are  described  in  more  than  general  detail  so 
that  a  viable  design  alternative  can  be  assured  and  selected.  This 
document  includes  descriptions  of  the  problem,  model,  data,  and  background 
information;  system  design  alternatives,  costs,  advantages  and  disadvan- 
tages of  each;  description  of  the  recommended  system  design  that  details 
assumptions,  limitations,  restrictions  and  expected  results;  software, 
hardware  and  interface  considerations;  overall  summary  of  the  major 
functions,  purpose,  data  requirements,  output  and  users;  critical  factors 
affecting  system  development;  system  development  plan  with  cost  and  per- 
sonnel by  task;  general  plan  of  action  for  management  and  organizational 
changes  and  decisions,  equipment  usage  and/or  acquisition,  personnel 
training,  and  user  participation  by  task  and  level  of  effort.  This 
document  is    both  an  historical    and   operational  one.    It  records  the 
process  by  which  the  computer  system  was  selected,  but  as  the  model  and 
related  elements  are  usually  modified  over  time,  it  will  have  to  be  amen- 
ded to  reflect  the  current  system  design. 

SOFTWARE  DESCRIPTION:     Software  Development  Phase 

The  documents  produced  here  are  the  ten  documents  described  in  the 
FIPS  PUB  38  Guidelines  (8).    These  documents  would  include  some  of  the 
information  developed  in  the  documents  described  above. 

VALIDATION  DESCRIPTION:    Validation  Phase 

This  report  includes  a  description  of  the  model  validation  plan 
agreed  to  by  the  user/sponsor  and  model  developer,  and  the  results  of 
implementing  the  plan.    Validation  of  the  model  must  include  tests  of  the 
model's  output  in  terms  of  comparisons  to  historical  data,  acceptability 
by  the  user  (experiential  or  intuitive  tests),  statistical  measures,  etc. 
The  developers  must  state  and  explain  the  deficiencies  and  divergences 
of  the  model's  output,  as  well  as  apparent  agreements.    The  validation 
report  should  delineate  the  problem  environment  in  which  the  model  is 
known  to  produce  results  acceptable  to  the  user/sponsor.    The  validity 
of  the  model  must  be  reestablished  whenever  the  model  is  changed. 
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TRAINING  PLAN:    Training  and  Education  Phase 


This  report  details  the  education  and  training  requirements  of 
the  project  and  describes  those  materials  that  need  to  be  produced, 
including  any  briefing  materials.    This  plan  should  describe  procedures 
for  turning  over  the  system  to  the  user  group  (if  different  than  the 
developer),  and  tests  for  ensuring  that  the  new  groups  understand  their 
aspect  of  the  model,  e.g.,  decision  makers  should  know  how  to  request 
computations  and  to  be  able  to  interpret  the  results.    The  outcomes  of  the 
training  and  education  effort  should  be  described. 

INSTALLATION  PLAN:    Installation  Phase 

This  report  describes  the  process  for  ensuring  that  the  verified 
and  validated  computer  model  is  installed  correctly  on  the  user's 
computer  system.    The  test  plan  is  described  and  the  results  recorded. 

IMPLEMENTATION  PLAN;    Implementation  Phase 

This  report  describes  the  process  by  which  the  computer  model 
is  made  a  part  of  the  user  organization,  who  are  responsible  for 
generating  and  requesting  model  analyses,  how  the  outputs  are  to  be 
used  in  informal  and  formal  reports,  security  of  the  system  and  its 
inputs  and  outputs,  final  authority  on  output  acceptance. 

MAINTENANCE  PLAN:  Maintenance  and  Update  Phase 

This  report  describes  the  processes  for  modifying  the  model  and 
its  data,  revalidation  and  who  are  responsible  for  the  updates.  Pro- 
gramming maintenance  would  be  included  in  the  software  documents. 

EVALUATION  PLAN:    Evaluation  and  Review  Phase 

Depending  on  the  objectives  and  needs  of  the  user,  an  evaluation 
or  periodic  review  process  is  described.    An  external  third-party  eval- 
uation plan  cannot  be  specified  by  the  user  or  developer,  but  is,  of 
course,  a  function  of  the  evaluator  team.    A  process  for  doing  this  is 
described  in  (60)  and  (61).    A  periodic  review  team  would  need  to 
develop  assessment  procedures  based  on  the  structure  of  the  model  and 
its  use„    Thus,  a  detailed    review  plan  cannot  be  developed  ahead  of 
time.    A  report  should  be  written  at  the  conclusion  of  the  evaluation  or 
review  that  describes  the  status  of  the  computer  model  and  recommendations 
for  its  change,  future  use  or  discontinuance. 
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DOCUMENTATION  PLAN:    Documentation  and  Dissemination  Phase 


This  report  describes  the  informal  and  formal  documents  to  be 
produced,  by  whom  and  during  what  phase(s)  of  the  project,  i.e.,  how  the 
documentation  and  dissemination  phase  is  to  be  managed.    The  report 
includes  a  statement  of  the  documentation  objectives,  dissemination  plan 
and  maintenance  and  updating  procedures. 

As  given  above,  the  relationship  between  the  documents  and  the 
model  life  cycle  phases  is  a  one  to  one  correspondence.    These  phases 
and  documents  can  be  combined  depending  on  the  computer  model  and  the 
needs  of  the  project.    Each  document  will  be  a  result  of  information  that 
is  recorded  in  project  notebooks,    studies,  programmer  logs,  memoranda, 
etc.    The  documentation  plan  spells  out  the  responsibilities  for  the 
recording  of  information  and  the  integrating  of  this  information  into 
the  project  documentation.    These  documents  will  describe  the  historical, 
developmental  and  operational  aspects  of  the  project  and  the  computer 
model.    However,  in  terms  of  the  complete  needs  of  users  and  analysts, 
the  above  set  of  documentation  is  incomplete.    What  is  required  is  the 
coalescing,  combining,  rewriting,  etc.  of  the  information  in  these 
documents  into  very  clear  and  readable  new  documents  termed  User's  Manual , 
Analyst ' s  Manual ,  and  Executive  Summary.    (Note  that  the  documents  of  the 
software  development  phase  include  programming  user,  operations  and 
maintenance  manuals.)    As  with  the  other  documents,  the  production  and 
maintenance  of  these  new  documents  are  described  in  the  documentation  plan, 
along  with  their  contents  and  assignments  for  their  development.  These 
documents  must  be  available  for  use  during  the  training  and  installation 
phases.    A  description  of  typical  contents  of  the  user's  and  analyst's 
manuals  are  given  in  Appendices  B  and  C.    Briefly,  we  note  the  following 
with  reference  to  these  three  documents. 

USER'S  MANUAL 

This  report  combines  information  from  the  other  project  documents 
to  represent  a  source  document  for  the  user  of  the  computer  model.    It  is 
to  provide  the  proposed  and  future  users  of  the  model  with  information 
necessary  to  use  it  effectively.    It  should  allow  non-programming  per- 
sonnel to  understand  the  workings  of  the  computer  model,  how  to  request 
runs  and  interpret  the  output. 

ANALYST'S  MANUAL 

This  report  combines  information  from  the  other  project  documents 
to  represent  a  source  document  for  analysts  who  have  been  and  will  be 
involved  in  the  development,  revisions  and  maintenance  of  the  computer 
model.    This  manual  should  include  only  those  technical  aspects  of  the 
model  that  are  essential  for  practical  understanding  and  application. 
The  detailed  technical  developments  will  be  contained  in  the  phase- 
produced  documentation,  e.g..  Model  Formulation  Description. 
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EXECUTIVE  SUMMARY 


This  report  is  an  essential  one  for  computer  models  used  in  a 
decision  environment.    It  is  directed  at  executives  of  the  organization 
who  will  be  required  to  interpret  and  use  the  results  of  the  computer 
model,  and  support  its  continued  use  and  maintenance.    Of  all  the  project 
documents,  this  one  has  to  be  carefully  and  clearly  written  in  order  to 
convey  the  full  impact  of  the  technical  developments.    The  report  should 
include  a  description  of  the  problem  setting  and  origins  of  the  project; 
a  general  description  of  the  model,  including  its  purpose,  objectives, 
capabilities  and  limitations;  the  nature,  interpretation,  use  and  restric- 
tions of  the  results  that  are  produced  by  the  model;  costs  and  benefits 
to  be  expected  in  using  the  model;  the  role  of  the  computer  model  in  the 
organization  and  decision  structure;  resources  required;  data  needs; 
operational  and  transfer  concerns;  and  basic  explanatory  material. 

The  final  document  required  to  be  produced  is  the  Model  Report. 

MODEL  REPORT 

This  report  is  a  nontechnical  summary  of  the  basic  information  that 
describes  the  computer  model.    Its  purpose  is  to  provide,  in  a  concise 
fashion,  a  description  of  the  computer  model  to  other  users  and  analysts 
so  they  may  determine  if  the  model  is  of  interest  to  them.    It  can  be 
included  in  other  documents  such  as  the  User's  and  Analvsts's  Manuals,  or 
distributed  separately.    A  typical  set  of  contents  is  that,  given  above 
in  Section  V  due  to  House  (42). 

It  should  be  clear  that  the  production  of  the  full  set  of 
documents  described  above  is  a  major  task  for  any  modeling  project. 
Project  managers  must  make  a  decision  early  in  the  model  life  cycle  as 
to  what  documentation  is  required  to  meet  the  objectives  of  the  project. 
Documentation  costs  can  be  quite  high.l/    But  these  costs  must  be  compared 
to  the  probable  additional  costs  if  adequate  documentation  is  not  prepared. 
As  noted  in  (35)  and  (46),  model  usage  and  transferability  are  direct  func- 
tions of  the  quality  and  quantity  of  documentation. 

It  is  hoped  that  the  material  presented  here  will  aid  model  users 
and  developers  in  advancing  the  cause  of  model  documentation.    Our  dis- 
cussion is  meant  to  serve  this  cause  and  thus,  contribute  to  improving 
the  development  and  use  of  computer  models. 


2/    Brewer  (64)  notes  that  many  computer  software  companies  spend  as  much 
money  on  documentation  as  for  the  software  itself.    In  (37),  Brewer 
also  suggests  the  possible  creation  of  a  new  professional  class  of 
documentation  specialists. 
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APPENDIX  A 
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1/    Adapted  from  (20). 
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ELEMENTS  OF  COMPUTER  MODEL  DOCUMENTATION 


I.  Background  Information 

A.  Model  name;  including  any  acronyms  or  short  titles. 

B.  Model  sponsors;  major  agency  and  direct  organizational 
unit  responsible  for  the  model's  development,  current 
owners  and  users  of  model. 

C.  Model  developers;  specific  groups,  agencies  and/or 
contractors  responsible  for  the  model  specifications 
and  its  development. 

D.  Model  project  director  and  other  responsible  personnel; 
including  government  contract  technical  monitor,  contrac- 
tor project  leader  and  other  government  and  contractor 
personnel  who  had  decision-making  responsibilities  in 
developing  the  model  and  their  areas  of  responsibility. 

E.  Model  abstract;  brief  description  of  the  purposes  of  the 
model,  its  objectives,  general  subject  matter,  methodolo- 
gical approach,  planned  utilization  in  a  decision-making 
environment,  designated  users  and  decision  makers. 

F.  Model  demographics;  model  start  and  completion  dates 
(approximately) ,  development  location(s),  current 
location(s)  of  mt)del,  model  cost,  relationship  to  other 
models. 

II.  Computer  Model  Documentation 

This  section  should  be  self-contained  and  of  sufficient  detail  so 
as  to  allow  the  model  user  or  an  evaluator  to  understand  the 
model  application  and  methodology  and  to  be  able  to  reproduce  the 
computing  operating  environment  and  the  results  of  the  test  case 
included  in  this  section.    The  documentation  should  also  enable 
the  user  or  evaluator  to  modify  data  inputs  and  run  the  program 
for  the  specified  ranges  of  parameters  and  extreme  cases.  This 
section  contains  the  following  information: 

A.  Model  name, 

B.  Abstract, 

C.  Introduction, 

The  objective  or  purpose  of  the  program,  with  any 
background  information,  such  as  feasibility  studies  or 
justification  writeups.    Summaries  that  describe  the 
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total  problem  environment  and  general  block  or  flow 
diagrams  should  be  included  here. 

D.  Problem  of  Task  Description, 

A  technical  description  of  the  problem  to  be  solved  or 
the  task  to  be  accomplished  and  formal  requirement 
specification,  when  provided,  should  be  included. 

E.  Method  of  Solution, 

For  scientific  applications,  this  includes  the  mathematical 
equations,  formulas,  and  technique  used  and  their  logical 
sequencing  within  the  program.    A  functional  flow  chart, 
such  as  a  block  diagram  or  a  logic  chart,  should  be  includ- 
ed to  provide  a  pictorial  display  of  this  logic.  For 
management  or  business  information  systems,  this  includes 
system  specifications,  logical  sequencing  of  events  within 
the  program,  and  again  the  functional  flow  chart. 

F.  Program  Description, 

The  minimum  contents  under  program  description  includes 
the  following: 

1.  Operating  Environment, 

Includes  hardware,  software,  (system  or  monitor, 
source  language)  machine  components  (memory 
requirements  and  tapes),  and  library  routines  used. 

2.  Program  Specifications, 

Includes  detailed  narrative  and  graphical  description 
of  the  programming  techniques  used  in  writing  the 
program;  i.e.,  calling  sequence,  overlay  structure, 
test  plan,  common  usage,  etc. 

3.  Subprogram  (Other  Than  Library), 

Any  subprogram  or  program  module  which  is  of 
such  a  general  nature  that  it  has  a  potential  of 
being  used  by  future  programs  should  be  documented 
in  detail  as  a  separate  entity.    It  can  then  either 
be  referenced  or  included  in  the  main  program  docu- 
mentation.   Special  emphasis  should  be  given  to  the 
model's  data  file  updating  and  editing  programs. 
Subprograms  which  have  little  or  no  sharing 
potential  should  be  documented  only  to  the  level  of 
detail  necessary  for  program  maintenance  and  modifi- 
cation.   The  level  of  detail  would  vary,  depending 
on  the  subprogram's  size  and  complexity.  Minimum 
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documentation  here  would  be  the  function  of  the  subprogram 
and  its  calling  sequence. 

4.  Source  Listing, 

Should  be  provided  or  readily  available. 

5.  Detailed  Flow  Chart, 

Detailed  flow  charts  should  be  provided. 

6.  Personnel  and  Program  Requirements, 

State  typical  personnel  time,  set  up  time,  and 

computer  time  required  to  make  a  typical  run  and 

to  analyze  the  output;  also,  types  of  skills  required. 


G.     Pror-^am  Use, 

A  description  of  the  procedure  required  to  set  up  and  run 
the  computer  program.    The  information  should  clearly  and 
adequately  describe  how  to  fill  out  the  data  sheets.  For 
a  terminal  interactive  system,  complete  logon,  logoff,  and 
instructions  for  interpreting  the  computer  requests  during 
operations  and  final  output  must  be  given.    A  sample,  annota- 
ted instructional  run  should  be  included.  ,For  a  batch  mode 
system,  instructions  must  be  given  as  to  how  to  submit  the 
system  and  data  deck,  with  all  its  program  forms,  and  how 
to  interpret  the  results  of  the  program.    The  following 
items  apply  to  both  a  batch  and  terminal  system,  except 
item  la  is  for  batch  and  lb  for  terminal. 

la.    Deck  Setup, 

An  illustration  of  the  deck/tape  setup  for  a 
computer  run,  showing  placement  of  control  cards, 
source  and/or  object  decks,  data,  restart  procedures, 
etc. 

lb     Terminal  Instructions, 

Logon,  logoff  procedures,  complete  set  of  computer 
queries  with  ranges  of  legal  responses,  data  input 
formats. 

2.  Input, 

File,  record,  and  data  element  descriptions  and 
formats,  including  the  origin  of  each  data  element. 
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3. 


Output, 


Files,  records,  data  element  descriptions  and 
formats,  in  addition  to  information  on  scratch  or 
intermediate  files. 

4.  Restrictions  and/or  Limitations, 

Hardware  and  software  restrictions,  data  ranges 
and  capacities,  program  behavior  when  restrictions 
are  violated,  and  recovery  procedures.    If  accuracy 
characteristics  are  significant,  they  should  be 
described  in  detail . 

5.  Editing  and  Diagnostics, 

Includes  the  "cause"  and  "cure"  of  program 
generated  diagnostic  messages;  all  checks  and 
balances  performed  to  ensure  accurate  and  complete 
output. 

6.  Test  Case, 

Includes  a  listing  of  input  and  output,  and  the 
machine  time  required  to  produce  this  output. 

7.  Data  and  Data  Files, 

Includes  description  of  the  data  files,  programs,  and 
procedures  for  updating  and  editing  the  data. 

H.  Symbols  and  Parameters, 

Define  all  meaningful  symbols  and  arrays  used  in  the 
routine  with  reference  to  the  mathematical  and/or 
technical  notations  and  terms  used  in  the  problem 
description.    Give  units  where  applicable.    If  possible, 
this  material  should  be  presented  in  a  tabular  form. 
Values  of  parameters  (e.g.,  a  computational  zero,  step 
sizes,  convergence  factors),  nominal  and  initial  values 
should  be  described,  along  with  their. ranges  and  discus- 
sion as  to  how  they  affect  the  computational  process. 

I.  Appendices, 
J.     References o 
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APPENDIX  B 


GUIDELINES  FOR  PREPARING  USER, 
PROGRAMMER  AND  ANALYST 
MANUALS  OF  COMPUTER  MODELSl/ 


1  /   From  (17). 
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I.     GUIDELINES  FUR  PREPARING  TllE  USEP.S  MANUAL 


A.  INTRODUCTION 

The  Users  Manual  is  the  source  docvjnent  for  the  user  cf  the 
program.     It  must  contain  sufficient  information  for  the  user 
to  thoroughly  understand  the  inner  workings  of  the  model  and 
to  accurately  use  the  model.     Occasionally  the  user  V7ill  be 
unfamiliar  with  data  processing  and  computer  techniques.  Care 
must  be  exercised  in  the  preparation  of  the  textual  material 
to  insure  that  a  layman  "non-progra:nmer"  can  understand  its 
contents . 

The  basic  text  of  the  Users  Manual  contains  four  sections: 
DESCRIPTION  OF  THE  MODEL 
INPUT  DATA  DESCRIPTION 
OUTPUT  DESCRIPTION 

INSTRUCTIONS  FOR  PREPARING  COMPUTER  RUNS 

The  contents  of  each  of  these  sections  are  described  in  the 
paragraphs  that  follov;. 

3.     SECTION  1  -  DESCRIPTION  OF  THE  MODEL 

The  description  of  the  model  will  include  the  following  sub- 
sections : 

GENERAL  DESCRIPTION 
DESCRIPTION  OF  METHODOLOGY 
LIMITATIONS  A.\T)  ASSUI^IPTIONS 

The  specific  information  that  will  be  included  in  each  sub- 
section is  as  follows: 

1.     GENERAL  DESCRIPTION 


Provide  a  complete  overview  of  the  model  including  its 
purpose,  intended  use,  general  magnitude  of  model  applicability, 
e.g.,  major  limits  and  assumptions,  inputs  and  outputs,  etc. 
Also  include  the  relationship  of  this  model  to  any  other 
models,   if  applicable;  e.g.,  model  X  prepares  the  input  data 
for  this  model.     This  overview  should  give  a  potential  user 
a  very  comfortable  relationship  v/ith  the  model. 
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2 .  METHODOLOGY 


Include  the  intimate  details  of  hov;  the  model  accomplishes 
its  intended  purpose.     These  details  should  be  provided  in  the 
sequence  in  which  they  are  performed  in  the  model,     Thiu  sec- 
tion will  contain  the  mathematical  formulas ,' derivations ,  proofs 
and  sufficient  detailed  description  to  determine  the  reasoning 
behind  the  approach  and  logic  of  the  model.     The  methodology 
subsection  will  provide  the  basis  for  substantiating  the  model 
results.     In  other  words,   the  description  of  the  logical 
manipulation  of  the  input  data  to  produce  the  model  results 
should  be  included.     A  gross  flov;  chart  of  che  logic  of  the 
program  should  also  be  included  to  assist  in  understanding  the 
program  design  m.ethodology . 

3.     LIMITATIONS  AND  ASSUMPTION'S 

The  overall  limits  of  the  model  concerning  the  magnitude 
of  the  items  considered  and  the  level  of  applicability  of  the 
model  should  be  included.     Since  the  assumptions  determine  the 
basis  for  the  program  design  and  establish  the  program  limits, 
a  complete  explanation  of  all  assumptions  will  be  provided. 
Assumptions  that  also  have  the  effect  of  restricting  the  pro- 
gram's utility  must  also  be  described. 

C.     SECTION  2  -  INPUT  DATA  DESCRIPTION' 

This  section  will  include  the  following  subsections: 

GENERAL  DESCRIPTION  OF  INPUT  DATA 
INDIVIDUAL  DATA  SET  DESCRIPTIONS 
DATA  SET  FORI-L^TS 

The  specific  information  that  will  be  included  in  each  subsec- 
tion is  as  follows: 

1.  GENERAL  DESCRIPTION  OF  INPUT  DATA 

This  subsection  will  include  a  description  of  the  overall 
data  structure,  the  nature  of  the  data  media  (e.g.,  tape  cards, 
etc.)  and  any  general  data  limitations.      It  should  also  incluc'e 
a  description  of  the  correlation  between,  data  types. 

2.  INDIVIDUAL  DATA  SET  DESCRIPTIONS 

Input  data  items  are  normally  organized  for  input  in  related 
groups,  such  as  aircraft  characteristics,     missile  charrctoris cics 
^tc. ,  or  as  the  data  items  that   are  input  on  one  punch  card. 
These  related  groups  of  data  establish  and  define  a  data  set  and 
should  be  described  together.     Each  data  set  description  should 
.begin  on  a  new  page.     Logically  related  data  sets  should  be 
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described  in  sequence.  The  information  that  should  be  included 
in  each  data  set  follov;s  and  is  illustrated  in  Figure  1,  Sample 
Data  Set  Description, 

-  DATA  SET  NAME  -  Include  the  name  and  or  acronym. 

-  DESCRIPTION  -  Include  an  overview  of  the  items  included 

in  the  data  set  and  the  purpose  and  function  of  the  data. 

-  MAXIMUM  NU:-IBER  OF  INPUTS  -  Include  the  maximiam  number  of 

inputs  of  this  type  that  can  be  prepared. 

-  CORRESPONDING  INPUTS  -  Specify  the  names  of  the  related 

data  sets  whose  data  items  are  dependent  upon  the  data 
input  values  for  this  data  set. 

-  DESCRIPTION  OF  DATA  SET  ITEMS  -  Include  a  detailed 

description  of  each  item  in  the  data  set  v;ith  the 
following  information:     item  name,  card  columns 
reserved  for  each  data  item,  the  minimum  and  maximum 
range  of  values  or  its  fixed  value,  and  a  definition 
of  the  item  to  include  the  use  of  the  item  in  the  .pro- 
gram,  its  relationship  to  other  items,  its  unit  of 
measurement,  source  (if  fixed) ,  and  any  information 
that  v;ill  assist  the  user  in  preparing  this  input. 
The  definitioh  will  also  include  the  requirements  for 
right-justified  columns   (i.e.,  right-most  character 
"    must  be  lined  up  'in  the  right-most  column  reserved 
for  the  item). 


1.  GENERAL  DESCRIPTION  OF  INPUT  DATA 

This  subsection  will  include  a  description  of  the  overall 
data  structure,   the  nature  of  the  data  media  (e.g.,  tape  cards, 
etc)   and  any  general  data  limitations.     It  should  also  include 
a  description  of  the  correlation  between  data  types. 

2.  INDIVIDUAL  DATA  SET  DESCRIPTIONS 

Input  data  items  are  normally  organized  for  input  in  related 
groups,   such  as  aircraft   characteristics,^    missile  characteristic 
ctc^ ,  or  as  the  data  items  that  are  i\iput  on  one  punch  card. 
These  related  groups  of  data  establish  and  define  a  data  set  and 
should  be  described  together.     Each  data  set  description  should 
.begin  on  a  new  page.     Logically  related  data  sets  should  be. 
described  in  sequence.     The  information  that  should  be  included 
in  each  data  set  follov;s  and  is  illustrated  in  Figure  1,  Sample 
Data  Set  Description. 
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DATA  SET  NAME  -  Include  the  name  and  or  acronym. 

DESCRIPTION  -  Include  an  overview  of  the  items  included 
in  the  data  set  and  the  purpose  and  function  of  the  data. 

MAXE-aiM  NUMBER  OF  INPUTS  -  Include  the  maximum  nuxTiber  of 
inputs  of  this  type  that  can  be  prepared. 

CORRESPONDING  INPUTS  -  Specify  the  names  of  the  related 
data  sets  whose  data  items  are  dependent  upon  the  data 
input  values  for  this  data  set. 

DESCRIPTION  OF  DATA  SET  ITEMS  -  Include  a  detailed 
description  of  each  item  in  the  data  set  v/ith  the 
following  information:     item  name,  card  columns 
reserved  for  each  data  item,  the  minimum  and  maximum 
range  of  values  or  its  fixed  value,  and  a  definition 
of  the  item  to  include  the  use  of  the  item  in  the  pro- 
gram, its  relationship  to  other  items,  its  unit  of 
measurement,  source  (if  fixed) ,  and  any  information 
that  v;ill  assist  the  user  in  preparing  this  input. 
The  definitioh  will  also  include  the  requirements  for 
right-justified  columns   (i.e.,  right-most  character 
must  be  lined  up  'in  the  right-most  column  reserved 
for  the  item),  the  use  of  leading  zeroes  (i.e.,  data 
item  will  be  padded  with  zeroes  to  the  left-most 
column  reserved  for  the  item  after  the  most  signifi- 
cant character  of  the  input  value)  and  the  use  or 
specification  of  required  or  implied  decimal  points. 
The  description  of  each  data  item  should  appear  in 
tabular  fom  as  illustrated  in  Figure  1. 
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INPUT:    Vehicle  Parameters 


Description:     This  input  specifies  the  weight,  empty  or  loaded, 
that  will  be  used  for  vehicles  in  a  basic  run  or  run  variation. 
This  input  is  not  optional  and  must  appear  in  each  run.  (It 
doesn't  have  to  be  re-input  for  each  run  variation  since  the 
first  setting  v;ill  be  used  for  each  run  variation  until  another 
input  of  its  type  is  encountered) . 

Maximum  Cards;     One  for  the  basic  run  and,  if  desired,  one  for 
each  run  variation. 


Corresponding  Inputs:  Vehicle  Characteristics 
Definition  of  Vehicle  Parameters: 


ITEM 


COLWINS       RANGE /VALUE 


DEFINITION 


ID 


1-14  "PARAMETERS 
VEH" 


Input  type 
Identifier. 


EMPTY 

WEIGHT 

OPTION 


19-24        Blank/Any  If  the  empty  weight 

Char,  of  vehicles  is  desired, 

then  put  any  nonzero 
character  in  any  of 
the  columns . 


LOADED 
WEIGHT 
OPTION 


25-30        Blank/Any  If  the  loaded  weight 

Char.  of  vehicles  is  desired, 

put  any  nonzero  charac- 
ter in  any  of  the 
columns . 


FIGURE  1 


-SAMPLE  DATA  SET  DESCRIPTION 
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DATA  SET  FORMATS 


In  addition  to  the  description  of  each  data  set,  a  format 
layout  or  creation  sheet  should  also  be  included.     This  ^ives 
the  user  a  visual  reference  for  preparing  the  input  data. 
Reference  Figure  2  for  a  Sample  Data  Set  Format. 

D.     SECTION  3  -  OUTPITT  DESCRIPTION 

This  section  will  include  the  following  subsections: 

GENERAL  DESCRIPTION  OF  OUTPUTS 
INDIVIDUAL  OUTPUT  DESCRIPTIONS 
SAMPLE  OUTPUT  F0R:L\T 

The  specific  information  that  will  be  included  in  each  subsec- 
tion is  as  follov;s: 

1.  GENER.^L  DESCRIPTION  OF  OUTPUTS 

Describe  the  overall  output  structure  to  include  types  of 
output,  relationship  of  one  output  type  to  another  and  the 
quantity  of  output  that  can  be  expected  under  varying  conditions 
of  input.     Also  include  v;hcther  or  not  certain  outputs  are 
optional  or  alv;ays  produced  for  each  computer  run. 

2.  TOIVIDUAL  OlTPbT  DESCRIPTIONS 

This  subsection  should  include  a  detailed  description  of 
each  output  report.     The  description  of  each  output  should  start 
at  the  beginning  of  a  page  and  the  sequence  of  the  output  pre- 
sentations should  be  provided  in  the  sequence  the  output  reports 
are  printed.     A  suggested  format  for  each  output  report  descrip- 
tion is  as  follows: 

-  OUTPUT  TITLE  -  Include  the  short  and  long  name. 

-  DESCRIPTION  -  Include  the  purpose  and  interpretation  of 

the  output  and  its  relationship  to  the  overall  results 
of  the  model. 
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-  DESCRIPTION  OF  OUTPUT  REPORT  ITEMS  -  Include  in  tabular 

fomi  the  name,  of  each  output  item  and  an  explicit 
description  of  each  item.     Also  include  any  information 
that  will  assist  in  validating  this  result. 

-  SAMPLE  OUTPUT  FOR>L\T  -  Output  formats  should  be  provided. 

These  can  be  actual  samples -of  the  model  output  or 

blank  formats  showing  the  output  headings  and  X's  showing 

where  the  data  would  appear. 

E.  SECTION  4  -  INSTRUCTIONS  FOR  PREPARING  COMPUTeR  RUNS 


This  section  of  the  Users  Manual  includes  the  following  subsec- 
tions: 

ORGANIZATION  OF  INPUT  DATA 
INSTRUCTION'S  FOR  SUBMITTING  RUNS 

The  information  to  be  included  in  each  of  these  subsections 
is  as  follov7s: 

1.  ORGANIZATION  OF  THE  INPUT  DATA 

This  subsection  must  include  the  organization  required  to 
input  the  data  for  executing  the  m.odel  as  well  as  any  optional 
or  alternate  organizations  that  allow  advantage  to  be  taken  of 
inherent  execution  flexibilities.     Both  a  narrative  explanation 
and  a  pictorial  representation  of  the  logical  organization  of 
the  input  data  will  be  provided.     Figure  3  contains  a  sample 
illustration  of  a  data  deck  organization.     Input  data  sets  that 
are  always  required  and  those  that  are  optional  must  also  be 
recapped  in  this  subsection. 

2.  INSTRUCTIONS  FOR  SUBMITTING  RUNS 

Normally,   to  execute  a  model,  control  cards  of  a  varying 
nature  are  required  in  addition  to  the  data  inputs.     An  explana- 
tion of  hov7  the  total  run  deck  is  to  be  set  up  must  be  provided 
including  the  details  of  how  to  prepare  the  control  cards.  A 
complete  listing  of  the  control  cards  in  their  proper  sequence 
with  the  card  column's  annotated  for  each  item  on  the  cards  must 
be  provided.     If  certain  control  cards  are  optional,  or  if  the 
run  can  be  set  up  in  more  than  one  way,  a  full  explanation  will 
be  given  for  each  option.     A  pictorial  representation  of  the 
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Integration  of  the  data  inputs  with  the  control  cards  must 
also  be  given. 

A  second  general  requirement  to  submit  a  run  for  execution 
is  the  preparation  of  a  work  request  form.     This  form  usually 
requests  run  specification  information  such  as  the  estimated 
execution  time,  estimated  lines  of  output,  input  and  output 
tape/disc  requirements,  core  storage  requirements,  computer 
language  the  model  is  written  in  and  the  computer  type  for  which 
it  was  written.     Therefore,  these  overall  run  specifications  must 
also  be  explained  in  detail.     If  the  execution  time  and  or  number 
of  lines  of  output  vary,  then  the  methods  for  estimating  these 
variations  should  be  provided.' 


II.     GUIDELINES  FOR  PREPARING  TIIE  PROGRAMMERS  MANUAL 
A.  INTRODUCTION 

The  primary  purpose  of  this  manual  is  to  serve  as  a  guide  for 
modifying  the  model.     The  guidance  must  be  provided  on  two 
levels,  the  macro  and  the  micro.    The  macro  level  includes  an 
overall  discussion  of  the  processing  techniques  used  and  their 
relationships.     The  micro  level  contains  a  -detailed  doscripcicn 
of  what  happens  in  each  routine.     The  macro  level  should  give 
the  programmer  a  grasp  of  how  everything   is  tied  together  and 
highlight  those  areas  of  particular'  vulnerability  to  change. 
The  micro  level  provides  the  factual,  details  about  what  occurs 
in  each  of  the  parts  of  the  model  on  an  individual  basis. 

A  secondary  purpose  of  this  manual  is  to  provide  useful  infor- 
mation for  the  programmer  v/ho  might  have  to  convert  the  model 
for  a  computer  other  than  the  one  for  which  the  model  was 
originally  written.     Peculiarities  of  the  model  design  that  are 
based  on  the  computer  hardv;are  and  or  system  softv/are  should  be 
discussed  V7ith  the  above  purpose  in  mind. 

This  manual  contains  four  major  sections: 

DESCRIPTION  OF  PROCESSING 

DESCRIPTION  OF  CO:iMOM  ARRAYS 

DESCRIPTION  OF  EACH  ROUTINE 

SOURCE  CODE  LISTING  07  THE  MODEL 

The  specific  contents  of  each  of  these  sections  will  be  described 
in  detail  as  follows. 
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B.     SECTION  1  -  DESCRIPTIOM  OF  PROCES S ING 


The  purpose  of  this  section  is  Co  provide  the  macro  view  of  the 
approach  to  the  model  design,  i.e.,  the  processing  techniques 
used.     The  information  that  is  noted  here  for  inclusion  can 
certainly  be  amplified  and  should  be  when  more  information  can 
be  provided  to  guide" a  programmer  who  has  never  heard  of  nor 
seen  the  model.     The  aim"  is  to  allow  the  programmer  to  modify 
the  model  properly,  so  that  a  change  to  one  routine  does  not 
lead  to  problems  in  other  routines    and  to  provide  for  a  smooth 
'iionversion  effort.     These  are  difficult  tasks  but  are  the  true 
purposes  of  this  section  of  the  Programmers  Manual. 

Th.3  Description  of  Processing  includes  the  following  subjec- 
tions: 

MODEL  SPECIFICATIOMS 
PROCESSING  TECHNIQUES 
GROSS  FLOW  CH^\RTS 

The  specific  information  that  should  be  included  in  each  of 
the  subsections  is  as  follov;s: 

1.     MODEL  SPECIFICATIONS 

Include  a  complete  rundo\\7n  of  the  computer  specifications 
as  follows: 

-  Computer  and  operating  system  for  which  the  model  was 

written. 

»•  Language  in  v;hich  the  model  was  written. 
JJumber  of  overlays  if  any. 

J^umber  of  routines  -  list  each  routine  and  provide  a 
hjrief  description  of  each  one  of  them. 

f  Mumber  of  core  locations. 

-  Execution  time. 

Input/output  device  requirements. 

«'  Upe  of  Standard/nonstandard  system/library  routines. 

Xf  there  are  any  peculiarities  about  any  of  the  above 
ppecif ications ,   they  should  be  discussed.     For  example,  if  the 
podel  was  written  in  the  FORTRAN  programming  language,  the  non- 
standard features  of  the  FORTRAN  compiler  that  are  peculiar  to 
the  operating  system  or  hardware  that  was  used  in  the  model 
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tievelopment  should  be  described.     This  subsection  is  really  of 
key  importance  to  the  programnier  who  has  to  convert  the  model 
for  another  computer.     Therefore,  the  more  detail  included 
concerning  the  model  specifications  the  better. 

•  2.     PROCESSING  TECHNIOU&S 

This  section  describes  the  program  processing  design 
specifications  from  the  vantage  point  of  post-model  develop- 
ment.    It  will  include  the  overall  approach  used  to  develop 
the  model,  highlighting  the  design  parameters  based  on  the 
hardware  or  system  softv;are  capabilities.     For  example,  since 
core  storage  in  the  G-635  computer  system  is  organized  in  36 
bit  fixed  v7ord  size  and  accomjr.odates  six  characters  per  v;ord , 
processing  the  input  data  of  a  model  may  be  organized  around 
this  word  configuration.     It  is  this  type  of  information  that 
should  be  included. 

•If  the  model  is  overlayed,  the  decisions  that  were  made 
to  determine  what  routines  could  be  overlayed  and  hov7  they 
were  to  be  overlayed  when  the  model  was  designed  should  be 
included.     Also  include  the  overall  control  flow  of  the  over- 
lays and  their  interaction  in  x^ord  or  flow  chart  form  or  both. 
If  the  model  is  not  overlayed,  describe  the  overall  processing 
flow.     Other  special  areas  that  should  be  covered  are  those 
of  initialization  and  wrap-up.     These  two  areas  are  generally 
overlooked,  but  deserve  attention  if  like  tasks  accomplished 
in  the  normal  processing  flow  are  done  in  a  different  manner 
in  these  two  areas.     Packed  or  unpacked  features  of  the  model 
data  should  be  described.     If  special  consideration  v;as  given 
to  determining  the  length  of  any  of  the  major  arrays,  those 
considerations  should  also  be  mentioned. 

3.     GROSS  FLOVJ  C FARTS 

These  charts  may  be  developed  as  a  part  of  the  previous 
subsection  or  be  located  in  one  central  location  in  the  manual. 
Gross-  flov7S  should  contain  operational  terminology,  not  the 
language  of  the  program.     They  are  more  meaningful  in  this 
form  to  a  programmer,  or  to  an  analyst  who  might  be  curious 
about ^the  actual  processing  techniques;  employed  by  the  model. 
Of  course,  routine/variable  names  should  be  used  with  a  brief- 
description  of  what  they  do  to  tie  the  charts  to  the  hard  cold 
code.     The  gross  floys  provide  an  overall  guide  for  walking 
through  the  code  to  shov;  the  interactive  process  flow  of  all 
routines  in  the  model.     They  should  also  depict  special  purpose 
areas  such  as  model  wrap-up  procedures  or  end-of-file  condicions 
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C.     SECTION  2  -  DESCRTPTTOM  OF  COMMON  ARRAYS 


All  arrays,  variables,  tables,  data  sets,  etc.,  that  are  shared 
by  more  than  one  routine  should  be  defined.     The  array  name 
should  be  provided  with  a  general  description  of  the  function 
of  the  array.     All  indices  should  be  defined  (both  implied  and 
specified) ,  follov;ed  by  a  definition  of  all  elements  in  the 
array.    A  list  of  the  routines  that  use  and/or  modify  this 
array  should  also  be  provided. 

D.     SECTION  .3  -  DESCRIPTION  OF  EACH  ROUTINE 

Each  routine  description  should  include  the  purpose  and/or 
function  of  the  routine,  i.e.,  its  role  in  the  overall  processii 
logic  of  the  model.     A  narrative  description  of  the  flow  is 
not  necessary,  and  should  only  be  included  if  it  amplifies  the 
code  and  points  out  any  subtleties  that  might  be  lying  therein. 
Flow  charts  are  also  optional,  but  are  helpful  when  the  routine 
is  very  large  and  v;hen  there  are  few  meaningful  comments  in 
each  routine. 


E.     SECTION  4  -  SQTJRCE  CODE  LISTING  OF  THE  MODEL 

If  the  model  or  routine  is  of  reasonable  size,  then  the  source 
listing  should  be  included  as  a  section  of  this  manual.     If  it 
is  large  and  unwieldy,  it , should  be  bound  separately  and  made 
available  upon  request. 

m.     GUIDELINES  FOR  PREPARING  THE  ANALYSTS  MANUAL 
A.  INTRODUCTION 

The  Analysts  Manual  is  the  source  document  for  the  Analysts 
using  the  model.     The  text  shall  be  factual,  concise,  clearly 
worded  and  illusrratod.     Sentence  form  will  be  simple  and 
direct.     Technical  knowledge  reflected  in  the  manual  should 
be  converted  into  the  most  easily  understood  wording  possible 
Discussions  of  theory  chall  be  omitced  except  where  essential 
for  practical  understanding  and  application.  Phraseology 
requiring  a  specialized  knowledge  of  computer  programiming  or 
computers  shall  be  avoided  except  where  no  other  wording 
will  convey  the  intended  meaning.     Where  necessary,  a 
"Glossary  of  Terms"  will  be  added  immediately  follovjing  the 
Table  of  Contents.     The  primary  emphasis  will  be  placed  upon 
the  specific  steps  to  be  follov/ed,   the  results  which  may  be 
expected  or  desired,  and  the  corrective  measures  required 
when  such  results  are  not  obtained. 


B-14 


The  basic  text  of  the  Analysts  Manual  contains  six  sections. 

MODEL  FUNCTION  DESCRIPTION 
DATA  COLLECTION  REQUIREMENTS 
INPUT  DATA 
OUTPUT  DATA 

■COMPUTER  RUN  ABORT  NOTES 
TECHNICAL  NOTES 

The  description  of  each  of  these  sections  are  described  in 
the  paragraphs  that  follow. 

B.     SECTION  1  -  MODEL  FUNCTION  DESCRIPTION 

The  description  of  the  model  will  contain  the  following 
subsections : 

A.  GENERAL 

B.  .  ASSUMPTIONS  AND  LII>IITATIONS 

C.  FLEXIBILITY 

The  specific  information  that  will  be  included  in  each  sub- 
section is  as  follows-: 

1.  GENERAL 

Provide  a  summary  description  of  the  model  including 
Its  purpose  and  intended  use.     This  overview  should  be 
sufficiently  detailed  to  give  the  analyst  a  very  comfortable 
understanding  of  the  model. 

2.  ASSUMPTIONS  AND  LIMITATIONS 

A  complete  identification  of  all  assvunptions  inherent 
in  the  model  will  be  made.     Each  will  be  discuGsed  with  the 
view  of  informing  the  analysts  of  the  restrictions  with  whic 
he  will  be  faced  in  using  the  model  and  in  interpreting  the 
outputs.     A  discussion  of  the  overall  limits  of  the  model 
on  the  magnitude  of  the  items  considered  and  the  level  of 
applicability  of  the  model  should  be  included. 

3.  FLEXIBILITY 

Provide  a  description  of  the  capability  for  adapting  the 
program  to  changing  requirements,  such  as  anticipated'  opera- 
tional changes,   interaction  with  new  or  improved  programs, 
and  planned  periodic  changes.     Components  and  procedures 
designed  to  be  subject  to  change  V7ill  be  identified.  Limita 
tions  on  the  flexibility  of  the  model  will  also  be  included. 


C.     SECTION  2  -  DATA  COLLECTION  REQUIREMENTS 
This  section  will  include  the  following  subsections: 
SCOPE 

INPUT  SOURCES  OF  DATA 
SUPPORT  PROGRAI^IS 
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The  specific  information  that  will  be  included  in  each  sub- 
section is  as  follows: 

1.  SCOPE 

This  subsection  V7ill  describe  the  types  of  information 
required  by  the  program  in  order  to  establish  the  data  values 
of  each  data  element.     It  shall  discuss,  as  a  minimum,  those 
types  of  information  needed  to  describe  the  data  element  in 
accordance  with  the  information  required  for  the  data  element 
library.     It  shall  also  specify  information  to  be  collected 
by  the  user,  logically  grouped  and  presented  in  a  manner  which 
will  enable  the  user  to  make  an  effective  response  to  the 
program's  requirements  for  data  elements. 

2.  INPUT  SOURCES  OF  DATA 

This  subsection  will  name  recommended  sources  from  which 
the  data  elements  should  be  gathered.    The  source  of  origin 
of  the  data  will  be,  for  example,  a  document,  an  organiza- 
tional unit,  etc.     Recommendations  as  to  whom  should  be 
responsible  for  providing  specific  data  inputs  will,  also, 
be  stated  here.     This  will  include  recommendations  regarding 
the  establishment  of  a  user  iiiput  reporting  organization,  if 
required.     Those  data  inputs  dependent  on  interfacing  systems^ 
unrelated  agencies,  or  specific  documents  should  be  the 
source  defined.     Specific  instructions  for  data  collection 
procedures  shall  be  given. 

3.  SUPPORT  PROGRAMS 

All  of  the  support  programs  available;  for  handling  the 
entry  of  data  into  the  data  base  will  be  discussed  briefly. 
Descriptions  shall  include  program  name,  functions,  and  major 
program  operating  considerations. 


D.     SECTION  3  -  INPUT  DATA 

This  section  will  include  the  following  subsections: 

DATA  ELEMEOT  PREPARATION 
DATA  BASE  DIPACTS 
ACCURACY  AND  VALIDITY 

The  specific  information  that  will  be  included  in  each  of 
the  subsections  is  as  follows: 

1.     DATA  ELD-fENT  PREPAPATION 

This  subsection  will  provide  a  detailed  description  of 
all  computer  program  inputs .     There  shall  be  a  description 
of  each  type  of  input  applicable  to  the  program.     Each  input 
shall  be  categorized  by  the  types  applicable  to  the  program 
(such  as  parameter,  data  type  A,  B,  etc.)  and  described  in 
detail  to  include  the  following,  as  applicable: 


B-16 


a.  Title  and  tag. 

b.  Format  and  acceptable  range  of  values. 

c.  Number  of  items. 

d.  Description  of  each  item  to  include  number  and  type 
of  characters'- (numeric ,  alpha,  decimal,  signed, 
unsigned),  range  of  values,  accuracy  requirements. 

e.  Means  of  entry  and  initiation  procedures;  e.g., 
typewriter,  card,  tape,  internal. 

f.  Flexibility,  such  as  capability  of  omitting  and 
adding  items . 

In  addition,  the  following  supplementary  information 
shall  be  given  for  each  data  element  where  applicable: 

a.  Critical  Value.     Many  elements  that  have  a 
range  of  values  wiiL  nave  one  value  that  is  particularly 
significant  to  the  analyst.     This  may  be  a  breakpoint,  a 
minimum  stock  level,  a  critical  wind  velocity,  etc.  When 
applicable,   the  critical  value  and  its  significance  to  the 
analyst  should  be  included. 

b.  Scales  of  Measurement.     For  numeric  scales,  if 
the  successive  steps  are  not  equal  to  "one"  on  the  units  of 
measurement,   then  the  increment  shall  be  specified.  For 
example,  a  data  element    representing  pressure  in  pounds  per 
Square  inch  (the  unit  of   measurement)  may  be  incremented  by 
pound,  half-pound,  or  ten-pound  intervals  .     For  numeric 
scales,   the  "scale  zero"  should  be  specified  if  it  is  not 
Implicit  in  the  units  of  measurement;  e.g.,  pressure  in 
pounds  per  square  inch  may  be  measured  relative  to  absolute 
zero  pressure  or  to  atmospheric  pressure.     For  numeric 
scales  where  averaging  functions  other  than  the  arithmetic 
mean  are  appropriate,   these  other  functions  (e.g.,  geometric 
mean,  root  mean  square,  harm.onic  mean)   should  be  indicated. 
For  non-numeric  scales,  any  relationships  indicated  by  the 
legal  values  should  be  sr.ated  if  not  ochert'/ise  specified. 
For  example,  a  code  indicating  lubricant  type  of  values  A, 
B,  C,'etc.,  should  show  whether  the  value  is  arbitrary 

(A  =  paraffin  oils,  B  =  graphite,  etc.)   or  whether  it 
Indicates  the  ordering  of  the  lubricants  by  some  parameter 
(A  =  viscosity  less  than  SAE  10,  B  =■-  viscosity  SAE  10  through 
SAE  40,  etc.)  . 

c.  Conversion  Factors.  Measured  quantities  that 
must  go  througn  analog  cigicai  conversion  processes  shall 
have  the  conversion  factors  specified. 

d.  Cross-References .     Many  input  values  are  so 
closely  related  tnac  a  cnange  in  one  value  should  cause  a 
change  in  another.     Describe  the  relationship  of  each  input 
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with  other  inputs,   i.e.,   if  the  input  value  of  A  is  changed, 
must  the  values  of  X,  Y,   or  Z  also  be  changed.     Build  a 
table  of  cross-references  for. each  input  value:  e.g. 


A  B  C 

X  -  X 

-  X  - 

X  -  X 


X 
X 


vhere  A,  B,  C  .   .   .  Z  are  data  item  names  and  an  X 
in  any  row  alerts  the  analyst  to  check  the  value  of  the  data 
item  indicated  for  the  column.     For  example,  A  might  be  an 
aircraft  name  and  C  might  be  itb  weight  and  Z  its  fuel  co- 
efficients . 

2.  DATA  BASE  I>TPACTS 

This  subsection  will  describe  the  impacts  associated 
with  collection  and  maintenance  of  che  data  base  on  equipment, 
software,  organizational,   operational,    and  developmental 
environments.     Impacts  on  the  a y stem  resulting  from  deficien- 
cies in  the  data  base  shall  also  be  given. 

3,  ACCURACY  AND  VALIDITY 

This  subsection  will  provide  a  description  of  accuracy 
requirements  imposed  on  the  system  (i.e.,   the  program  and 
computer  system.)  .     The  following  accuracy  requirements  must 
be  considered: 

a.  Accuracy  requirements  of  mathematical  calculations. 

b.  Logical  and  legal  accuracy  of  alphanumeric  data. 

c.  Accuracy  of  transmitted  data, 
E,     SECTION  4  -  OUTPUT  DATA 

This  section  V7ill  include  the  following  subsections: 

OUTPUT  DESCRIPTION 
OUTPUT  FORiM/DEVICE 
ACCURACY  OF  OUTPUT 

The  specific  information  that  will  be  included  in  each  sub- 
section follows: 
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1.     OUTPUT  DESCRIPTION 


This  subsection  will  provide  a  detailed  description  of 
all  program  outputs.     A  description  of  each  type  of  output 
applicable  to  the  program  will  be  included.     Each  output 
item  will  be  categorized  by  type  (such  as  data  type  A,  B, 
etc.)   and  described  in  detail  to  include  the  follov;ing  as 
applicable : 

a.  Title  and  tag. 

b.  Format  to  include  headings,  line  spacing,  arrange- 
ment, totals,  etc. 

F.     SECTION  5  -  COMPUTER  RUN  ABORT  NOTES 

This  section  will  include  the  following  subsections: 

ABORT  CODES  AND  MEANING 
WARNING  NOTICES  AND  ^rEi\NING 

The  specific  information  that  will  be  included  in  each 
subsection  follows: 

1.  ABORT  CODES  AND  >TEi\NING 

This  subsection  v/ill  identify  all  O/S  System  abort  codes 
as  v;ell  as  codes  built  into  the  program.     The  meaning  of 
the  code  will  be  clearly  defined.     Action  to  be  taken  by 
the  analyst  (including  consultation  with  Programmer  or 
Operator)  will  be  noted. 

2.  WARNING  NOTICES  AND  >TEANING 

This  subsection  v/ill  identify  all  O/S  warning  notices  as 
well  as  warning  notices  put  out  by  the  program.     The  meaning  of 
each  notice  and  possible  effects  on  the  operation  of  the  program 
and  on  outputs  frora  the  program  v/ill  be  clearly  identified. 
A  notation  v;ill  be  made  as  to  whether  or  not  the  warning 
notice  can  be  safely  ignored  by  the  analyst. 


G.     SECTION  6  -  TECHNICAL  NOTES 

This  section  v;ill  describe  all  algorithms  in  the  m.odel 
necessary  for  the  analyst's  understanding  of  how  the  computer 
program  uses  the  input  data  to  calculate  information  reflected 
in  the  output.     For  example,  the  method  of  calculating  the 
fuel  consumption  of  the  aircraft  will  be  fully  described. 
Other  examples  include;     description  of  algorithm  used  for 
vectoring  an  interceptor  against  an  intruder,  description  of 
algorithm  used  for  calculating  impact  points  of  weapons, 
etc.     Input  values  used  in  calculations  will  be  specifically 
identified  by  name.     Headings  will  be  used  for  each  algorithm 
such  as  "Fuel  Consumption  Calculations",  "Weapon  Impact  Point 
Calculations",  etc.     As  many  headings  as  necessary  will  be 
used  and  will  be  of  the  following  form: 


B-19 


1,  FUEL  CONSUMPTION  CALCULATION 

2,  WEAPON  IMPACT  POINT  CALCULATIONS 
etc. 

Depending  upon  the  length  of  this  section,   it  may  be  structured 
as  a  separate  volume,  or  it  (or  its  subsections)  may  be  struc- 
tured as  an  appendix(ces)   to  this  volume. 

c.  Number  of  items. 

d.  Description  of  each  item  to  include  number  and 
type  of  characters  (alpha,  numeric,  symbol,  etc.) , 
range  of  possible  values. 

e.  Data  selection  criteria  will  be  presented  to 
establish  the  basis  for  selecting  information  for 
display. 

f.  Description  of  plots  or  graphic  displays  will 
include  the  coordinates  used,  symbols  used,  type 
of  graphic  technique  (i.e.,  points  or  continuous) , 
number  of  curves  per  sheet,  etc. 

g.  Means  of  display;  e.g.  ,  CRT,  printer,  typev7riter, 
projector,  plotter. 

h.  Length  of  output,  including  special  handling 
requirements  due  to  variations  in  length  (such 
as  pages  of  printout,  feet  of  paper  tape,  etc.). 

2.  OUTPUT  FORM/DEVICE 

Output  data  elements  may  be  presented  to  the  user  sym- 
bolically, graphically,  or  may  be  used  as  input  to  some  other 
automated  system.     If  the  user  is  to  receive  a  sensible 
presentation,   the  description  should  specify  whether  the 
user  will  receive  the  data  element  as  part  of  a  hard  copy 
printout,  a  symbol  in  a  CRT  display,  a  line  on  a  drawing,  etc. 
Any  limitation  on  the  presentation  due  to  the  form  of  display 
will  be  discussed. 

3.  ACCUR-\CY  OF  OUTPUT 

This  subsection  will  discuss  the  accuracy  of  the  output 
values  in  relation  to  the  method  of  their  derivation.  A 
reviev;  of  the  algorithms  used  for  their  derivation  may  be' 
necessary  at  this  point.     Any  limitations  on  the  use  of  the 
data  by  the  analyst  should  be  discussed.     Any  peculiarities 
of  the  system   that  may  result  in  erroneous  output  (such  as, 
overflows  causing  loss  of  significant  digits  in  some  tables) 
should  be  noted.     Corrective  actions  to  be  taken  by  the 
analyst  .in  the  cases  noted  should  be  described  (i.e.,  should, 
he  contact  the  programmer  to  modify  the  program,  or  can  he 
modify  the  program  or  modify  his  input  data  to  take 
care  of  the  problem) .     Any  corrective  action  described  must 
be  clearly  stated  so  that  there  is  little  chance  for  error 
on  the  part  of  the  analyst. 
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APPENDIX  C 


DETAILED  TABLES  OF  CONTENTS 
FOR  USER,  PROGRAMMER  AND  ANALYST 
MANUALS  OF  COMPUTER  MODELS  1/ 


1/   From  (39). 
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USER'S  MANUAL 
TABLE  OF  CONTENTS 


SECTION  DESCRIPTION 
1.0 


INTRODUCTION;    ALLOWS  ' NON- PROGRAMMER' 
USER  TO  UNDERSTAND  INNER  WORKINGS 
OF  PROGRAM  AND  ACCURATELY  USE  IT 


2.0  DESCRIPTION  OF  MODEL 

2.1  General  Description  (Overview) 

2.1.1  Purpose 

2.1.2  General  Model  Applicability 

2.1.3  Major  Limits  and  Assumptions 

2.1.4  Inputs  and  Outputs 

2.2  Methodology 

2.2.1  How  Model  Does  It 

2.2.2  Math  Formulas,  Derivations,  and  Proofs 

2.2.3  Reasoning  and  Logic 

2.2.4  Gross  Flow  Chart 

2.3  Limitations  and  Assumptions 

2.3.1  Allowable  Magnitudes 

2.3.2  Level  of  Applicability 

2.3.3  Explanation  of  Assumptions 

2.3.4  Restricting  Assumptions 
3.0  INPUT  DATA  DESCRIPTION 

3.1  General  Description  of  Input 

3.1.1  Overall  Data  Structure 

3.1.2  Data  Media 

3.1.3  General  Data  Limitations 

3.1.4  Correlation  Between  Data  Types 
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USER'S  MANUAL  -  TABLE  OF  CONTENTS  (continued...) 


SECTION  DESCRIPTION  PAGE 

3.2  Individual  Data  Set  Description 

3.2.1  Data  Set  Name 

3.2.2  Description 

3.2.3  Maximum  Number  of  Inputs 

3.2.4  Corresponding  Inputs 

3.2.5  Description  of  Items 

3.2.6  Data  Set  Format 
4.0  OUTPUT  DESCRIPTION 

4.1  General  Description  of  Output 

4.1.1  Overall  Output  Structure 

4.1.2  Relationships  Between  Output  Types 

4.1.3  Amount  of  Output 

4.1.4  Optional  Output 

4.2  Individual  Output  Description 

4.2.1  Output  Title 

4.2.2  Description  of  Table 

4.2.3  Description  of  Items 

4.2.4  Sample  Output 
5.0  RUN  INSTRUCTION 

5.1  Organization  of  Input 

5.1.1  Narrative 

5.1.2  Pictorial 

5.2  Submitting  Runs 

5.2.1  Control  Cards  (JCL?)  and  Total  Run  Deck 

5.2.2  Optional  Cards 

5.2.3  Run  Cards  and/or  Work  Request  Form 
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PROGRAMMER  MANUAL 
TABLE  OF  CONTENTS 


SECTION  DESCRIPTION  PAGE 

1.0  ALLOWS  PROGRAT^MER  TO; 

1.1  Modify  program,  and 

1.2  Convert  it  for  Other  Computers 

2.0  DESCRIPTION  OF  PROCESSING 

2.1  Model  Specifications 

2.1.1  Computer  and  0/S 

2.1.2  Language 

2.1.3  Overlays 

2.1.4  Routines,  Brief  Descriptions,  Size,  Entries 

2.1.5  Core  Storage 

2.1.6  Execution  Times 

2.1.7  I/O  Device  Requirements 

2.1.8  Use  of  Standard/NonStandard  Library  Routines 

2.2  Processing  Techniques 

2.2.1  Overall  Approach,  Gross  Flow  Charts  in  Operational 
Terminology 

2.2.2  Design  Parameters  -  Hardware/Software,  Packing 

2.2.3  Overlay  Rationale 

2.2.4  Overlay  Control  Flow,  Interaction  -  Flow  Chart 

2.2.5  Initialization 

2.2.6  Wrap-Up 

3.0  INTRODUCTION  -  DESCRIPTION  OF  COMMONS 

3.1  Common  A 

3.2  Common  B,  etc. 

4.0  DESCRIPTION  OF  COMMON  ARRAYS  AND  VARIABLES 
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SECTION  DESCRIPTION  PAGE 

4.1  Arrays,  Variables  -  indices,  routines  that 
Use/Modify 

4.2  Tables 

4.3  Data  Sets 

5.0  DESCRIPTION  OF  EACH  ROUTINE 

5.1  Purpose,  Function,  Role,  Flow  Charts  Optional 
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ANALYST  MANUAL 
TABLE  OF  CONTENTS 
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SIMULATION  TODAY 

The  Society  for  Computer  Simulation  (Simulation  Councils,  Inc.)  P.O.  Box  2228,  La  Jolla,  Ca.  92037 


SIMULATION:  FROM  ART  TO  SCIENCE  FOR 
SOCIETY 

by 

John  McLeod 

ABSTRACT 

Simulation  can  be  a  more  effective  tool  for  the  study  of 
problems  of  our  times  if  the  practitioners  will  adopt  pro- 
cedures which  will  make  simulation  more  a  science  than 
an  art.  A  big  step  in  that  direction  would  be  the  develop- 
ment and  use  of  a  standard  format  that  would  assure  ade- 
quate and  uniform  documentation.  Such  a  format  is  pro- 
posed, and  criticism  is  solicited. 

INTRODUCTION 

The  fact  that  simulation  today  is  still  more  an  art  than  a 
science  is  unfortunate  for  many  who  could  use  simulation 
to  advantage.  Although  the  "man  in  the  street"  doesn't 
know  it  matters  (he  is  in  no  mood  to  trust  either  simu- 
lation or  science),  it  would  be  well  if  we  practitioners  of 
the  art  start  applying  more  scientific  principles  in  our 
work. 

Simulation,  even  in  its  current  state  of  development,  is 
important.  And  it  is  becoming  more  so.  There  is  no  more 
powerful  tool  to  help  us  understand  the  current  and  future 
problems  of  our  society.  However,  so  long  as  practitioners 
of  the  art  continue  to  do  their  own  thing  in  their  own  way, 
and  report  their  work  in  such  a  way  that  it  is  difficult  or 
impossible  for  others  to  check  or  repeat  their  work — or 
don't  report  it  at  all — simulation  will  remain  an  art.  That 
may  be  all  right  for  the  artist,  but  it  does  nothing  to  inspire 
our  decision-makers  to  have  confidence  in  us.  Before  we 
can  sell — and  not  oversell — simulation  to  those  who  can 
use  it  to  help  alleviate  the  problems  of  our  society,  we 
must  take  steps  to  discourage  simulationists  from  riding  off 
in  all  directions.  This  will  require  better  communication 
among  all  concerned.  By  better  communication,  I  don't 
mean  just  more  comprehensive  communication,  but  more 
comprehensible  communications.  Complete  and  uniform 
documentation  will  be  a  prime  requisite. 

The  documentation  of  the  development  of  a  model  and  of 
simulation  runs  should  be  complete  enough  to  allow 

The  reporter's  peers  to  evaluate  his  work 

Others  to  repeat  the  experiment 

Others  to  build  on  the  work  reported  instead  of 

having  to  repeat  it — or  start  over  in  their  own  way. 

This  is  not  a  new  idea.  This  writer  and  others  have  made 
efforts  in  that  direction  in  the  past.^  Some  have  listened, 
but  most  simulationists,  independent  thinkers  that  they 
are,  have  continued  to  build  individualized  models  to  suit 
their  own  purposes  (and  too  often,  consciously  or  other- 
wise, their  own  egos).  This  tendency  is  detrimental  to  our 
long-range  interests.  If  simulation  is  to  become  a 
respected  and  accepted  tool  for  the  study  of  important 
problems,  it  must  be  made  more  of  an  organized 


discipline.  We  know  how  to  wield  one  of  the  most 
powerful  tools  that  technology  has  placed  at  the  disposal 
of  society.  I  think  we  should  take  our  responsibility  for  its 
further  development  and  proper  use  very  seriously. 

I  was  obviously  groping  for  a  more  professional  way  of 
reporting  our  work  when  I  wrote  that  article  in  1970' 
urging  uniform  documentation.  We've  come  a  long  way 
since  then,  but  I'm  still  groping,  still  looking  for  a  better 
way  in  a  field  that  is  developing  fast — but  not  with  as  much 
discipline  as  it  should  have.  Proper  documentation  of  a 
simulation  project  will  establish  a  benchmark,  a  point  of 
departure  from  which  one  can  determine  how  far  he  has 
progressed,  and  to  which  he  can  return  if  he  gets  lost. 

I  would  now  like  to  update  my  thinking  of  more  than  three 
years  ago  and  propose  a  more  complete  format  for  docu- 
mentation. 

OUTLINI: 

The  documentation  format  should  cover  the  following 
items  describing  the  overall  project,  each  model 
developed  under  the  project,  and  each  simulation  for 
which  each  model  is  used. 

1.  Project  information 

1.1  Project  title 

1.2  Responsible  organization 

1.3  Contact 

1.4  Project  objective 

1.5  Project  duration 

1.6  Funding 

1.6.1  Source 

1.6.2  Amount 

1.6.3  Period 

2.  Model  development  information 

2.1  Name  of  model 

2.2  Name  of  modeler(s) 

2.3  Purpose  for  which  model  was  developed 

2.3.1  Specific 

2.3.2  General 

2.4  Discipline(s)  involved 

2.4.1  Primary 

2.4.2  Supporting 

2.5  Data  requirements 

2.6  Method  of  development 

2.7  Assumptions 

2.8  Cost  of  development 

2.9  Availability 

2.9.1  To  developer 

2.9.2  To  others 

2.10  Compatibility 

2.10.1  Computer  system  used  in  development 
2.*0.2  Other  systems  used 
2.10.3  Language(s) 

2.11  Extent  of  use 

2.11.1  By  developer 

2.11.2  By  others 
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3.  Description  of  model 

3.1  Classification  of  model 

3.1.1  Focus 

3.1.2  Scope 

3.1.3  Sophistication 

3.2  Block  diagram  of  system  modeled 

3.3  Program  flow  or  wiring  diagram  of  model 

3.4  Notation 

3.5  Validation 

3.6  Reference  information 

3.7  Distinctive  features 

3.8  Antecedents  of  model 

3.9  Current  related  models 

4.  Simulation(s) 

4.1  Title 

4.2  Purpose 

4.3  Assumptions 

4.4  Experimental  design 

4.5  Data  requirements 

4.6  Data  used 

4.7  Run  time 

4.8  Cost  per  run 

4.9  Results 

4.10  Justifications  of  assumptions 

4.11  Analysis 

5.  Discussion 

5.1  Comments 

5.2  Conclusions 

6.  Literature 

6.1  Project  reports 

6.2  References 

6.3  Bibliography 

Obviously,  it  would  be  foolish  to  expect  rigorous 
adherence  to  any  such  format.  Some  of  the  information 
requested  may  be  considered  "sensitive,"  while  in  other 
cases  it  might  not  be  of  sufficient  pertinence  to  warrant 
inclusion.  And  in  some  instances  the  information  may 
simply  not  be  available.  But  if  simulationists  will  use  the 
proposed  format — or  something  like  it  which  others  might 
propose — as  a  checklist  to  aid  in  the  preparation  of  more 
complete  and  more  uniform  documentation,  it  will 
certainly  go  a  long  way  toward  making  simulation,  if  not  a 
science,  at  least  a  more  lespectable  technology. 

SUGGESTIONS  FOR  USE  OF  OUTLINE 

1.  Project  information 

1.1  Project  title 

This  should  be  the  title  of  the  overall  project  of 
which  the  modeling  or  simulation  may  be  just  a 
part. 

1.2  Responsible  organization 

This  is  the  name  and  address  of  the  organization 
responsible  for  the  overall  project.  If  the  project  is 
supported  by  an  external  source  (e.g.,  by  a  grant), 
this  would  be  the  organization  responsible  for  the 
money  and  equipment  furnished. 

1.3  Contact 

This  is  the  person  or  persons  to  contact  for  further 
information.  If  the  address  is  other  than  that  given 
in  1.2,  the  full  mailing  address  should  be  given.  The 
organizational  mail-stop  or  code  and  telephone 
number  and  extension  should  also  be  given. 


1.4  Project  objective 

This  is  the  objective  of  the  overall  project,  which 
may  be  of  greater  scope  than  that  of  the  modeling 
or  simulation  to  be  covered  later. 

1.5  Project  duration 

Give  date  that  project  was  established  and 
expected  completion  date. 

1.6  Funding 

1.6.1  Source 

Give  name  of  funding  organization. 

1.6.2  Amount 

Give  total  dollars.  If  equipment  is  con- 
tributed, list  major  items  or  estimate  value. 

1.6.3  Penod 

Give  dates  covered  by  support  listed  in  1.6.2 

2.  Model  development  information 

2.1  Name  of  model 

This  might  be  the  computer-callable  name  of  the 
program.  If  an  acronym,  spell  it  out  (e.g., 
WLDREC:  WorLD  RECycling  model). 

2.2  Name  of  modeler(s) 

2.3  Purpose  for  which  model  was  developed 
"The  same"  here  will  indicate  the  same  as  1.4. 

2.3.1  Specific 

Give  reason(s)  for  developing  model  (e.g.,  to 
test  hypothesis  that .  .  .). 

2.3.2  General 

Give  other  uses  to  which  the  model  has  been 
or  might  be  put  (e.g.,  to  study  other 
problems  related  to  .  . .). 

2.4  Disciplines  involved 

These  need  not  be  fields  of  endeavor  recognized 
as  distinct  disciplines  (e.g.,  economics),  but  may  be 
more  descriptive  of  the  work  (e.g.,  land  use). 

2.4.1  Primary 

Give  the  discipline(s)  that  the  model  was  pri- 
marily developed  to  serve  (e.g.,  inter- 
national relations). 

2.4.2  Supporting 

List  other  disciplines  required  in  the  develop- 
ment of  the  model,  preferably  in  descend- 
ing order  of  importance. 

2.5  Data  required 

Give  kind  of  data  (e.g.,  population)  and  source 
(e.g.,  census). 

2.6  Method  of  development 

Give  method  of  development  (e.g.,  theoretical, 
empirical,  other). 

2.7  Assumptions 

List  all  assumptions  concerning  both  data  and 
causality  that  led  to  the  model's  being  developed 
in  the  way  it  was. 

2.8  Cost  of  development 

Give  actual  or  estimated  total  cost  of  the  model 
and  what  the  cost  includes. 
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2.9  Availability 

2.9.1  To  developer 

Is  the  model  operative?  What  will  it  take  to 
make  it  operative? 

2.9.2  To  others 

Is  the  model  proprietary  or  classified?  Can  it 
be  obtained  by  others?  How?  In  what  form 
(e.g.,  computer  listing,  deck,  paper  or 
magnetic  tape,  other)?  What  are  the  charges? 

2.10  Compatibility 

2.10.1  Development  of  computer  system 

On  what  equipment  was  the  model 
developed? 

2.10.2  Other  systems 

On  what  other  computer  systems  has  it 
been  or  might  it  be  run  with  negligible 
change? 

2.10.3  Language(s) 

In  what  language(s)  was  the  program 
written?  Is  it  available  in  others? 

2.11  Extent  of  use 

2.11.1  By  developer 

What  actual  use  has  been  made  of  the 
model  by  the  developer?  What  use  is 
planned? 

2.11.2  By  others 

Has  the  model  been  used  by  others?  By 
whom?  To  what  extent? 


3.  Model  description 

3.1  Model  classification 

What  kind  of  model  is  it?  How  is  it  run — batch  or 
interactively?  Locally  or  remotely?  Is  the  com- 
puter time-shared? 

3.1.1  Focus 

Give  the  primary  fields  of  interest  that  the 
model  serves  (e.g.,  political  science,  resource 
usage,  etc.). 

3.1.2  Scope 

Give  entity  modeled  (e.g.,  an  industrial  plant, 
a  river  basin,  the  U.S.  Senate,  etc.). 

3.1.3  Sophistication 

Where  does  the  model  fit  in  the  "Fuzz  to 
Fact"^  spectrum  (e.g.,  preliminary  studies, 
evaluating  alternatives,  predicting  the 
future)? 

3.2  6/oclc  diagram  of  system  modeled 

This  should  have  a  block  for  each  component  of 
the  real-world  system  modeled,  and  show  lines 
between  the  blocks  indicating  the  causal  relation- 
ships of  the  components  as  well  as  exogenous 
inputs  and  outputs. 

3.3  Program  or  wiring  diagrams 

A  program  flow  diagram  should  be  shown  in  the 
case  of  digital  models,  a  wiring  diagram  in  the  case 
of  analog,  and  both  in  the  case  of  a  hybrid  model. 


3.4  Notation 

A  complete  description  of  the  notation  used  in  3.2 
and  3.3,  as  well  as  any  narrative  description,  should 
be  included  here.  The  notations  and  definitions 
must  be  carefully  checked  for  consistency 
throughout  the  documentation. 

3.5  Validation 

Describe  how  the  model  was  validated. 

3.6  Reference  information 

This  should  be  a  computer  listing  of  the  program 
and  the  output  of  a  standard  check  run  for  a  digital 
model,  a  plot  of  a  standard  check  run  for  an 
analog,  or  both  in  the  case  of  a  hybrid  model.  It 
should  give  all  "numbers"  used  to  set  up  the  run, 
and  be  annotated  in  such  a  way  that  either  the 
developer  at  a  future  date,  or  another  user  of  the 
model,  can  make  sure  that  if  the  model  is  to  be 
rerun  or  used  by  someone  else,  he  will  be  working 
with  the  model  that  he  thinks  he  is  working  with. 

3.7  D/s(/nct/ve  features 

How  does  the  model  differ  from  related  models? 
How  is  it  better?  What  are  its  limitations?  What  are 
the  possible  pitfalls  that  might  be  encountered  in 
its  use? 

3.8  Model  antecedents 

Have  similar  models  been  built  before?  If  so,  give 
proper  credit.  Is  the  current  model  a  follow-on  or 
a  distinct  "mutant"? 

3.9  Current  relations 

Do  other  models  exist  that  have  the  same  or  a 
closely  related  purpose?  How  does  this  model 
relate  to  them?  Are  they  another  attempt  to  solve 
the  same  problem,  or  can  the  results  be  expected 
to  be  complementary,  i.e.,  to  present  two  aspects 
of  a  larger  problem?  Are  there  possibilities  of 
online  interconnection  and  interaction  between 
the  models? 

4.  Simulation(s) 

A  simulation  will  be  taken  to  mean  an  experiment 
performed  on  a  model  instead  of  the  real-world  simu- 
land.  Multiple  runs  using  the  same  experimental 
design  may  be  considered  one  simulation.  However,  if 
the  design  of  the  experiment  or  the  procedure  is 
changed,  it  should  be  considered  another  experiment 
and  items  4.1  through  4.11  should  be  covered  again. 

4.1  Title 

This  should  be  descriptive  of  the  simulation 
experiment,  and  may  be  made  up  of  the  model 
name  plus  a  subtitle  (e  g.,  "WLDREC:  Effect  of  cost 
of  recycling"). 

4.2  Purpose 

This  is  the  purpose  of  the  individual  experiment. 

4.3  Assumptions 

List  assumptions  made  in  the  design  of  the 
experiment. 

4.4  Experimental  design 

This  should  give  the  procedure  to  be  followed  in 
the  experiment,  step  by  step. 
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4.5  Data  requirements 

Give  the  data  requirements  for  the  individual 
simulation  run(s)  that  differ  from  those  for  the 
model's  reference  run  (item  3.6). 

4.6  Data  used 

This  might  best  be  a  computer  listing  of  those  lines 
of  data  that  differ  from  the  reference  run. 

4.7  Run  time 

This  should  be  total  as  well  as  mainframe  time. 

4.8  Cost  per  run 

This  should  be  given  for  both  mainframe  and 
peripherals. 

4.9  Results 

This  can  be  "raw  data"  (e.g.,  a  computer  printout) 
and  plots,  graphs,  etc.,  prepared  by  hand  or 
machine. 

4.10  Justification  of  assumptions 

This  is  most  important,  and  should  be  done  before 
any  analysis  of  the  results  is  attempted.  Assump- 
tions that  influenced  the  development  of  the 
model  as  well  as  those  related  to  the  specific  simu- 
lation experiment  should  be  considered. 

4.11  Analysis 

Describe  conclusions  drawn  from  the  results,  and 
give  reasoning  where  the  conclusions  are  not 
obvious. 

Discussion 

5.1  Comments 

Add  any  comments  here  that  might  further  illum- 
inate aspects  of  the  project  not  covered  else- 
where or,  if  preferred,  give  a  brief  narrative 
description  for  the  benefit  of  the  casual  reader. 

5.2  Cofic/us/ons 

Relate  development  of  the  model  and  the  simula- 
tion experiments  to  the  overall  project  objective 
given  in  1.4. 


6.  Literature 

6.1  Project  reports 

List  reports,  presentations,  and  articles  generated 
by  the  project. 

6.2  References 

List  publications  actually  referred  to  in  the  docu- 
mentation. 

6.3  Bibliography 

List  publications  which  influenced  the  work  docu- 
mented or  which  are  closely  related  to  it. 


CONCLUSION 

I  realize  that  just  scanning  the  foregoing  list  of  suggested 
items  might  stop  some  potential  users  before  they  start.  If 
so,  they  should  fill  out  the  easy  parts,  then  look  at  the 
blanks.  For  each  part  they  have  skipped,  they  should  ask 
themselves.  Is  this  information  necessary  for  a  person 
unfamiliar  with  the  details  of  my  work  to  understand  it^  Is 
it  information  that  would  be  necessary  to  reconstruct  the 
model  and  rerun  the  simulation^  if  the  answers  are  No,  the 
information  may  be  left  out  with  impunity.  If,  however,  the 
answer  to  either  question  is  Yes,  then  every  effort  should 
be  made  to  give  the  necessary  information.  On  the  other 
hand,  if  more  information  than  is  called  for  by  this  format 
would  be  necessary  to  repeat  the  experiment,  then  that 
information  should  be  added. 

I  will  appreciate  suggestions  for  additions,  or  justifiable 
deletions,  or  just  plain  comments. 
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